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Abstract 

Guidance  from  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  (USD(AT&L))  requires  100  percent  of  defense  programs  to  incorporate  cost  as 
an  independent  variable  (CAIV)  and  evolutionary  acquisition  (EA)  plans  within  their 
management  baselines.  Historically,  these  two  concepts  have  been  implemented 
independent  of  one  another.  In  reality,  CAIV  and  EA  are  tightly  coupled.  Integration  of 
these  two  initiatives  enables  warfighters  and  developers  to  better  allocate  constrained 
resources,  respond  to  fluctuations  in  program  funding,  and  plan  for  future  development 
activities. 

This  research  creates  a  decision  tool  to  assist  the  DoD  acquisition  community  in 
satisfying  the  intent  of  the  USD(AT&L)  guidance.  Using  multiattribute  design 
evaluation  techniques,  a  core  CAIV  model  is  formulated.  Next,  the  core  model  is 
expanded  to  incorporate  the  dominant  features  of  EA.  The  expanded  model  seeks  to 
optimize  overall  utility  across  a  horizon  of  multiple  development  increments. 
Additionally,  technical  risk  factors  are  integrated  to  discount  the  realized  level  of 
attainment  for  design  attributes.  Using  a  DoD  command  and  control  system  development 
as  the  case  study,  the  fully  fonnulated  CAIV/EA  model  is  implemented  and  in  a  PC 
spreadsheet.  An  optimization  application  solves  the  mathematical  program  for  a  series  of 
cost  constraints.  The  resulting  data  are  collected  and  translated  into  a  variety  of  graphics. 
Sensitivity  analysis  is  performed  to  understand  the  response  caused  by  variations  in  the 
model’s  parameters.  Model  limitations  are  discussed  and  recommendations  for  further 
investigation  are  presented. 
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INTEGRATING  COST  AS  AN  INDEPENDENT  VARIABLE  ANALYSIS  WITH 


EVOLUTIONARY  ACQUISITION  -  A  MULTIATTRIBUTE  DESIGN  EVALUATION 

APPROACH 


I.  Introduction 


Background 

On  November  27,  2001  the  newly  appointed  Under  Secretary  of  Defense  for 

Acquisition,  Technology,  and  Logistics  (USD(AT&L)),  the  Honorable  E.C.  Pete 

Aldridge  delivered  testimony  to  the  Commission  on  the  Future  of  the  U.S.  Aerospace 

Industry.  Enacted  under  Section  1092  of  the  Floyd  D.  Spence  National  Defense 

Authorization  Act  for  Fiscal  Year  (FY)  2001,  the  Commission  was  formed  to  study  the 

future  of  the  United  States  aerospace  industry  in  the  global  economy,  particularly  in 

relationship  to  United  States  national  security  (Heuttner,  2001).  With  over  $60  billion  in 

defense  related  procurement,  $40  billion  in  research  and  development  efforts,  and  another 

$40  billion  in  services,  spares,  and  logistics  support,  the  Under  Secretary  did  not 

embellish  when  he  stated,  “My  office  has  a  significant  impact  on  the  direction,  health, 

and  operations  of  the  aerospace  industry”  (Aldridge,  2001). 

According  to  the  Commission’s  charter,  its  mission  was  broadly  stated: 

The  Commission  shall  develop  and  recommend  a  series  of  public  policy 
reforms  which  will  pennit  the  U.S.  aerospace  industry  to  create  superior 
technology,  excel  in  the  global  marketplace,  profit  from  investments  in 
human  and  financial  capital,  benefit  from  coordinated  and  integrated 
government  decision-making,  assure  our  national  security,  access  modern 
infrastructure,  and  give  the  United  States  a  capacity  throughout  the  21st 
century  to  reach  for  the  stars  (Heuttner,  2001). 

Pursuant  this  cause,  at  the  hearing  Aldridge  presented  his  “Five  Goals.”  The 
Under  Secretary  testified,  “I  believe  (these  five  goals)  will  have  a  direct  effect  and 
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significant  influence  on  the  outcome  of  your  task”  (Aldridge,  2001).  Briefly,  Aldridge’s 
stated  goals  for  USD(AT&L)  were  as  follows: 

•  Improve  the  credibility  and  effectiveness  of  the  acquisition, 
technology,  and  logistics  support  process. 

•  Revitalize  the  acquisition,  technology,  and  logistics  workforce. 

•  Improve  the  health  of  the  defense  industrial  base. 

•  Rationalize  our  weapon  systems  and  infrastructure  with  the  new 
defense  strategy. 

•  Initiate  those  high  leverage  technologies  that  will  provide  the 
warfighting  capabilities  and  strategies  of  the  future  (Aldridge,  2001). 

Early  the  following  year,  in  a  memorandum  to  the  service  secretaries,  Aldridge 
stated,  “In  order  to  guide  and  measure  our  progress  toward  accomplishing  these  goals,  I 
have  established  a  set  of  metrics,  some  of  which  I  plan  to  report  on  to  the  Secretary  of 
Defense”  (Aldridge,  2002a:  1).  The  initial  set  of  metrics  approved  by  the  Under  Secretary 
pertained  to  the  first  goal:  improving  the  credibility  and  effectiveness  of  the  acquisition, 
technology,  and  logistics  support  process.  At  the  Aerospace  Commission  hearing  the 
previous  November,  Aldridge  provided  additional  detail  on  this  goal: 

•  Too  many  cost  overruns,  schedule  delays,  and  perfonnance  failures  have 
destroyed  our  credibility  in  the  eyes  of  the  Congress.  Their  Constitutionally 
mandated  responsibility  for  oversight  and  our  lack  of  credibility  leads  to  the 
inevitable  micromanagement  of  our  acquisition  processes; 

•  Cycle  times  are  too  long  and  the  logistics  support  system  cannot  yet 
meet  the  standards  we  see  for  support  of  commercial  systems; 

•  We  are  far  too  optimistic  in  performance,  cost  and  schedule  when  we 
make  budget  requests  and  we  simply  must  do  a  better  job  of  being 
more  realistic  in  our  estimates,  even  if  that  means  we  cannot  start  as 
many  programs;  and 

•  Reducing  cycle  time,  more  realistic  cost  estimating,  spiral 
development  to  reduce  risk  and  time,  controlling  requirements  creep, 
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and  interoperability  mandates,  are  examples  of  things  we  can  do  to  re¬ 
establish  our  credibility,  and  our  ability  to  manage  efficiently  and 
effectively  (Aldridge,  2001). 

Through  the  memorandum,  Aldridge  explained  how  he  intended  to  meet  his  first 
goal:  “I  have  approved  a  metric  to  require,  by  the  end  of  FY02,  100  percent  of  defense 
programs  to  incorporate  a  cost-as-an-independent  variable  (CAIV)  plan  and  to  have  an 
evolutionary  acquisition  (EA)  or  spiral  development  implementation  plan  in  place” 
(Aldridge,  2002a:  1).  The  memorandum  goes  on  to  explain  the  Department  of  Defense 
(DoD)  5000  series  (mandatory  acquisition  guidance)  would  be  adjusted  during  their  next 
update  cycle  to  reflect  these  new  program  management  requirements  (Aldridge,  2002a:  1). 

This  guidance  is  significant  because  it  represents  the  first  instance  where  the 
concepts  of  CAIV  and  EA  are  cited  together  in  a  mandatory  acquisition  directive.  While 
neither  of  the  two  are  new  initiatives  (they  both  appeared  during  the  acquisition  reform  of 
the  mid-  to  late-nineties),  historically  they  have  been  addressed  and  implemented 
independent  of  one  another.  To  completely  understand  the  ramifications  of  this  guidance, 
it  is  important  to  have  a  clear  understanding  of  CAIV  and  EA. 

Cost  as  an  Independent  Variable  (CAIV) 

CAIV  is  a  DoD  strategy  that  makes  total  life-cycle  cost,  as  projected  within  the 
acquisition  environment,  a  key  driver  of  system  requirements,  performance 
characteristics,  and  schedules.  Simply  put,  CAIV  treats  cost  as  a  military  requirement. 
This  is  a  conceptual  change  in  thinking  from  the  days  of  requirement-,  performance-,  and 
sometimes  schedule-driven  costs  (Rush,  1997:161). 

In  1995,  Under  Secretary  of  Defense  for  Acquisition  and  Technology  Dr.  Paul 
Kaminski  launched  a  DoD-wide  working  group  to  address  approaches  and  measures  to 
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reduce  and  control  weapon  system  life  cycle  costs.  CAIV  is  a  result  of  this  endeavor. 

The  working  group  summarized  their  findings: 

This  strategy  entails  setting  aggressive,  realistic  cost  objectives  for 
acquiring  defense  systems,  and  managing  risks  to  obtain  these  objectives. 

Cost  objectives  must  balance  mission  needs  with  projected  out  year 
resources,  taking  into  account  existing  technology  as  well  as  high- 
confidence  maturation  of  new  technologies.  This  concept  has  become 
known  as  “cost  as  an  independent  variable”  (CAIV),  meaning  that,  once 
the  system  performance  and  objective  cost  are  decided  (on  the  basis  of 
cost-performance  trade-offs),  the  acquisition  process  will  make  cost  more 
of  a  constraint,  and  less  of  a  variable,  while  nonetheless  obtaining  the 
needed  military  capability  of  the  system  (Kaminski,  1995:3). 

Buried  within  this  definition  is  the  central  tenet  of  the  CAIV  approach:  an  increased  role 

for  the  end-user  through  participation  in  setting  and  adjusting  program  goals  throughout 

the  program,  particularly  in  the  cost-performance  trade-off  process.  Beyond  the 

definition,  the  working  group  also  generated  a  conceptual  approach  to  implement  CAIV 

processes  within  defense  acquisition  programs.  This  approach  is  characterized  by  the 

following  aspects: 

•  Setting  realistic  but  aggressive  cost  objectives  early  in  each  acquisition 
program. 

•  Managing  risks  to  achieve  cost,  schedule,  and  performance  objectives. 

•  Devising  appropriate  metrics  for  tracking  progress  in  setting  and 
achieving  cost  objectives. 

•  Motivating  government  and  industry  managers  to  achieve  program 
objectives. 

•  Putting  in  place  for  fielded  systems  additional  incentives  to  reduce 
operating  and  support  costs  (Kaminski,  1995). 

These  guidelines  summarized  Dr.  Kaminski’s  policy  and  strategy  to  develop  and  field 

affordable  weapons  systems  that  are  responsive  to  user  needs. 
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To  the  casual  observer,  CAIV  should  not  appear  as  a  revolutionary  idea.  The 
prudent  consumer  only  buys  what  he  or  she  can  afford.  In  our  private  lives,  we  constrain 
our  personal  acquisitions  within  our  available  budgets.  We  make  trade-offs  between 
vacations  and  car  payments,  dinners  out  and  purchases  at  the  supermarket,  etc.  We  also 
look  for  ways  to  save  money  (clipping  coupons,  carpooling).  All  of  these  activities 
mirror  the  CAIV  guidelines  cited  above.  Unfortunately,  prior  to  the  release  of  the  CAIV 
working  group  report  in  1995  and  the  incorporation  of  its  recommendations  into  the  DoD 
5000  series  in  1996,  this  line  of  thinking  did  not  permeate  the  acquisition  management 
community  (Rush,  1997:162).  As  was  previously  mentioned,  defense  system 
acquisitions  have  traditionally  been  driven  by  requirements  and  perfonnance. 

It  is  also  important  to  note  the  concepts  embodied  within  CAIV  are  not  unique  to 
the  DoD  environment.  Around  the  same  time  Dr.  Kaminski  and  the  CAIV  working 
group  was  preparing  to  release  its  guidance,  the  Consortium  for  Advanced  Manufacturing 
International  (CAM-I)  published  a  book  entitled,  Target  Costing:  The  Next  Frontier  in 
Strategic  Cost  Management.  Target  costing  in  the  commercial  sector  is  analogous  to  the 
public  sector’s  CAIV.  While  CAIV  is  a  strategic  process  concerned  with  managing 
aggressive  cost  objectives  (within  authorized  budgets),  target  costing  is  a  strategic  profit 
and  cost  management  process  focused  on  managing  the  allowable  amount  of  cost  that  can 
be  incurred  on  a  product  while  still  earning  the  required  profit  from  the  product. 

To  clearly  describe  the  commercial  counterpart  to  CAIV,  CAM-I  provides  a  concise 
definition: 

The  target  costing  process  is  a  system  of  profit  planning  and  cost 
management  that  is  price  led,  customer  focused,  design  centered,  and 
cross-functional.  Target  costing  initiates  cost  management  at  the  earliest 
stages  of  product  development  and  applies  it  throughout  the  product  life 
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cycle  by  actively  involving  the  entire  value  chain  (Ansari  and  Bell, 
1997:3). 


The  similarities  between  the  two  processes  are  readily  apparent.  Both  place  the 
end-user  as  their  primary  focus.  Additionally,  CAIV  and  target  costing  are  concerned 
with  establishing  cost  targets  and  then  making  design  trade-offs  early  in  the  life  of  a 
project.  Finally,  risk  is  managed  throughout  the  lifecycle  so  targets  (i.e.,  aggressive  cost 
objectives)  are  met.  The  concepts  of  target  costing  have  penneated  private 
manufacturing  sectors.  According  to  Toyota’s  annual  report  for  1993,  “Cost 
management  is  going  to  be  for  the  automobile  industry  in  the  1990’s  what  quality  control 
was  in  the  1970s  and  ‘80s”  (Ansari  and  Bell,  1997:5).  Since  embracing  best  commercial 
practices  is  a  cornerstone  of  DoD  acquisition  refonn,  it  is  not  surprising  USD(AT&L)  has 
mandated  CAIV  be  implemented  across  all  defense  system  programs. 


Evolutionary  Acquisition  (EA) 

EA  and  spiral  development  (SD),  are  two  terms  continually  misused  and 

misinterpreted  by  the  acquisition  community.  This  impression  is  substantiated  by  the 

memorandum  released  by  Aldridge  on  April  12,  2002.  In  the  memo  Aldridge  states: 

“Since  the  publication  of  DoD  Directive  5000.1  and  DoD  Instruction 
5000.2,  in  which  the  Department  established  a  preference  for  the  use  of 
EA  strategies  relying  on  spiral  development,  there  has  been  some 
confusion  about  what  these  terms  mean  and  how  spiral  development 
impacts  various  processes  such  as  contracting  and  requirements  generation 
that  interface  with  an  EA  strategy.  The  purpose  of  this  memorandum  is  to 
address  those  questions”  (Aldridge,  2002b:  1). 

Aldridge  provides  a  clear,  concise  definition  of  these  terms  and  explains  the  interrelations 

between  the  concepts. 
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EA  is  an  acquisition  strategy  that  defines,  develops,  produces  or  acquires,  and 
fields  an  initial  hardware  or  software  increment  (or  block)  of  operational  capability.  This 
strategy  is  based  on  technologies  demonstrated  in  relevant  environments,  time-phased 
requirements,  and  demonstrated  manufacturing  or  software  deployment  capabilities. 
These  capabilities  can  be  provided  in  a  shorter  period  of  time,  followed  by  subsequent 
increments  of  capability  over  time  that  accommodate  improved  technology  and  allow  for 
full  and  adaptable  systems  over  time.  Each  increment  will  meet  a  militarily  useful 
capability  specified  by  the  user  (i.e.,  at  least  the  thresholds  set  by  the  user  for  that 
increment);  however,  the  first  increment  may  represent  only  60  to  80  percent  of  the 
desired  final  capability  (Aldridge,  2002b:  1). 

According  to  the  USD(AT&L)  definition,  there  are  two  basic  approaches  to  EA. 

In  one  approach  the  ultimate  functionality  can  be  defined  at  the  beginning  of  the 
program,  with  the  content  of  each  deployable  increment  detennined  by  the  maturation  of 
key  technologies.  In  the  second  approach,  the  ultimate  functionality  cannot  be  defined  at 
the  beginning  of  the  program,  and  each  increment  of  capability  is  defined  by  the 
maturation  of  the  technologies  matched  with  the  evolving  needs  of  the  user.  In  both 
cases,  an  increment  is  considered  a  militarily  useful  and  supportable  operational 
capability  that  can  be  effectively  developed,  produced  or  acquired,  deployed,  and 
sustained.  Each  increment  of  capability  will  have  its  own  set  of  thresholds  and  objectives 
set  by  the  user  (Aldridge,  2002b:  1). 

Often,  the  terms  EA  and  SD  are  used  interchangeably.  The  memorandum 
attempts  to  delineate  between  the  two  by  providing  a  separate  definition  for  the  later.  SD 
is  an  iterative  process  for  developing  a  defined  set  of  capabilities  within  one  increment. 
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This  process  provides  the  opportunity  for  interaction  between  the  user,  tester,  and 
developer.  In  this  process,  the  requirements  are  refined  through  experimentation  and  risk 
management,  there  is  continuous  feedback,  and  the  user  is  provided  with  the  best  possible 
capability  within  the  increment.  Each  increment  may  include  a  number  of  spirals.  Spiral 
development  implements  EA  (Aldridge,  2002b). 

Integrating  CAIV  and  EA 

The  brief  review  of  the  concepts  of  CAIV  and  EA  reveals  there  is  cause  for  Under 
Secretary  Aldridge  requiring  managers  of  defense  acquisitions  to  generate  corresponding 
plans  for  their  respective  programs  (as  expressed  in  the  January  19,  2002  memorandum). 
CAIV  and  EA  are  tightly  coupled.  The  most  apparent  linkage  is  the  role  the  user  plays 
in  each.  Within  CAIV,  the  user  is  a  pivotal  player  in  the  cost-requirements-perfonnance 
trade-off  process.  Additionally,  as  the  fiduciary  advocate  for  the  program  (the  one  who 
submits  budget  requests  into  the  DoD  planning,  programming,  and  budgeting  system 
(PPBS)),  the  user  must  also  participate  in  the  creation  of  aggressive  cost  objectives. 

From  an  EA  perspective,  the  user  must  define  the  system’s  core  and  incremental 
capabilities.  Additionally,  the  user  must  describe  the  threshold  and  objective  levels  of 
performance  for  these  capabilities.  All  of  these  activities  are  dependent  upon  one 
another.  Changes  made  to  capabilities  create  ripples  affecting  cost.  Aggressive  cost 
objectives  and  their  ensuing  trade-offs  have  profound  effects  upon  the  system’s 
capabilities,  its  schedule,  and  what  is  ultimately  delivered  to  the  user. 
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Research  Questions 


The  guidance  provided  in  the  January  19,  2002  USD(AT&L)  memorandum 
explicitly  requires  program  managers  to  create  separate  implementation  plans  for  CAIV 
and  EA  respectively.  However,  because  of  the  apparent  connection  between  the  two 
activities,  the  challenge  of  creating  two,  independent  plans  is  futile.  Any  perturbation 
made  to  one  impacts  the  other.  This  scenario  begs  the  question,  “Is  it  possible  to  develop 
a  process  that  integrates  CAIV  objectives  with  the  EA  framework?”  If  so,  this  process 
would  enable  users  and  developers  to  better: 

•  Allocate  constrained  resources, 

•  Respond  to  fluctuations  in  program  funding,  and 

•  Plan  for  future  development  activities  (i.e.,  increments). 

This  research  endeavors  to  create  a  process/model  to  assist  program  managers,  cost 
analysts,  engineers,  and  users  to  meet  the  first  goal  set  by  Under  Secretary  Aldridge: 
achieving  credibility  and  effectiveness  in  the  acquisition  and  logistics  support  process. 
Along  the  way,  this  research  will  explore  the  following  questions: 

1 .  How  might  one  generate  and  graphically  depict  the  relationship  between  system 
cost  and  performance  for  a  defense  program? 

2.  What  is  the  marginal  benefit  (or  detriment)  to  a  weapon  system’s  performance 
given  an  increase  (or  decrease)  in  funding  beyond  a  cost  objective? 

3.  How  might  one  optimally  allocate  resources  across  a  program  planning  horizon 
spanning  several  increments? 

Research  Overview 

This  chapter  has  explored  the  underlying  requirement  for  a  process  that  integrates 
CAIV  with  EA.  USD(AT&L)  has  stated  all  defense  programs  must  have  plans  for  each. 
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However,  creating  plans  independent  of  one  another  will  most  likely  not  meet  the  intent 
of  the  Under  Secretary’s  goal.  Ultimately,  a  new  model  is  necessary.  This  research  will 
explore  the  development  of  such  a  model  using  a  notional  Air  Force  ground  based 
command  and  control  (C2)  system  as  a  test  case. 

Chapter  II  develops  more  complete  definitions  for  both  CAIV  and  EA.  The 
chapter  also  explores  their  foundations  in  DoD  acquisition  guidance.  Next,  a  survey  of 
popular  approaches  used  to  implement  these  initiatives  (independent  of  one-another)  is 
presented.  Finally,  some  time  is  spent  reviewing  candidate  analytical  techniques  for  use 
in  the  formulation  of  the  CAIV/EA  model. 

Chapter  III  introduces  the  methodology  used  to  create  an  integrated  CAIV/EA 
model.  First,  the  core  CAIV  model  is  formulated.  Following  this  fonnulation,  the  model 
is  expanded  to  incorporate  features  associated  EA.  Finally,  potential  strategies  for  model 
evaluation  and  analysis  are  discussed. 

Chapter  IV  integrates  the  model  developed  in  the  previous  chapter  with  the 
notional  ground  based  C2  system.  The  characteristics  of  the  notional  C2  system  are 
applied  to  the  model.  The  model  is  then  completely  implemented  and  exercised.  The 
results  from  these  activities  are  collected  and  analyzed.  Finally,  the  behavior  of  the 
CAIV/EA  model  is  evaluated  through  the  use  of  sensitivity  analysis. 

Chapter  V  summarizes  the  outcomes  of  the  research  questions  explored.  The 
chapter  also  presents  the  limitations  of  the  research.  Finally,  the  chapter  presents 
opportunities  for  further  study  on  this  subject. 
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II.  Literature  Review 


Overview 

Research  by  RAND  identifies  two  dominating  feature  of  the  modern  U.S.  market 
for  weapons  and  weapons  systems: 

•  First,  it  is  characterized  by  a  single  buyer,  the  DoD,  which  defines  the  product 
and  controls  the  sales  opportunities  of  weapon  system  providers; 

•  Second,  it  is  distinguished  by  a  higher  degree  of  technical  complexity  and 
innovation  than  most  commercial  markets  (Lorell  et  ah,  2000:13-14). 

With  regards  to  this  first  feature,  the  weapons  market  model  clearly  diverges  from  a 

commercial  market  model;  where  diverse  and  autonomous  buyers  choose  products 

offered  by  competitive  sellers  on  the  basis  of  their  price  and  perfonnance  characteristics. 

The  second  feature  compounds  the  differences.  Developers  of  new  weapons  systems 

frequently  push  the  limits  of  known  technology,  incorporating  designs  and  materials  that 

are  largely  unproven.  In  contrast,  most  commercial  product  developers  tend  to  improve 

incrementally  on  existing  technologies  (Lorell  et  ah,  2000:14). 

In  the  mid-1990s,  the  problems  of  declining  defense  budgets  and  growing 
weapons  system  procurement  costs  lead  some  government  and  industry  officials  to 
advocate  the  integration  of  the  U.S.  military  and  civilian  industrial  bases,  a  concept 
commonly  referred  to  as  Civil-Military  Integration  (CMI)  (Lorell  et  ah,  2000:  1). 
Advocates  of  CMI  attributed  the  aforementioned  problems  to  the  unique  features  of  the 
U.S.  weapons  markets.  They  believed  that  DoD  adoption  of  commercial  business 
practices  and  a  more  commercial-like  market  structure  would  spur  the  development  of 
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high-performance  weapon  systems  at  lower  costs  than  could  be  achieved  under  the 
current  heavily  regulated  acquisition  process  (Lorell  et  ah,  2000:  2). 

The  current  round  of  acquisition  refonn  (AR),  begun  early  in  the  Clinton 
administration,  has  made  CMI  a  centerpiece  (Lorell  and  Graser,  2001:  3).  Two  initiatives 
closely  linked  with  CMI  are  EA  and  CAIV.  This  chapter  begins  with  a  discussion  of  EA 
and  the  methods  used  to  implement  this  strategy.  Next,  CAIV  analysis  and  the 
techniques  available  for  its  execution  are  reviewed.  Finally,  the  chapter  presents  a  brief 
overview  of  Utility  Theory  and  its  application  to  decision  problems  characterized  by 
multiple  decision  attributes. 

Implementing  EA 

To  summarize  the  definition  provided  in  the  previous  chapter,  EA  is  characterized 
by: 

•  Incremental  delivery  of  operational  capability, 

•  Time  phased  requirements  based  upon  technological  maturity  and 
availability  of  resources, 

•  Shorter  cycle  times,  and 

•  Adaptable,  open  systems  (Aldridge,  2002b:  1). 

While  this  definition  helps  to  provide  an  initial  mental  picture  of  EA,  more  detail  is 
necessary  to  completely  describe  the  concept. 

It  is  valuable  to  understand  the  initial  conditions  which  led  to  the  genesis  of  EA. 
The  Joint  Logistics  Commanders  Guidance  for  Use  of  EA  to  Acquire  Weapon  Systems 
was  published  in  1987  (with  a  re-issue  in  1998)  in  response  to  “A  clearly  discernable 
need  to  reduce  the  time  necessary  to  field  (weapons)  systems  -  a  need  driven  by  the  rapid 
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acceleration  in  technologies  used  in  such  systems”  (DSMC,  1998:vii).  This  document 
cites  the  results  of  two  major  studies1,  which  found  the  use  of  the  standard  acquisition 
approaches  (described  in  Department  of  Defense  Directives  (DoDD)  and  Instructions 
(DoDI))  have  often  led  to  unsatisfactory  results  (DSMC,  1998:2-1).  As  the  studies 
revealed,  these  difficulties  arose  primarily  because  it  was  often  “impossible  to  define 
detailed  operational  capabilities  or  functional  characteristics  for  the  complete  system 
before  undertaking  full  scale  development”  (DSMC,  1998:2-1).  Additionally,  whenever 
the  development  effort  is  begun  without  clear  definition  of  system  operational  concepts, 
capabilities,  and  functional  characteristics,  “It  is  very  likely  that  the  development  process 
will  be  long,  costly,  and  unstable.  Consequently,  the  developed  system  will  be 
unsatisfactory  and  logistically  unsupportable” 

(DSMC,  1998:2-1). 

External  pressures  stimulated  the  need  to  change  the  standard  DoD  acquisition 
approach  as  well.  These  pressures  are  political,  economic,  and  technological  in  nature: 

•  The  emphasis  on  the  European  continental  threat,  the  Soviet  Union,  has 
been  replaced  by  multiple  and  constantly  changing  threats; 

•  A  fiscally  constrained  economy  results  in  fewer  new  system  starts,  more 
emphasis  on  modifications  to  current  systems,  and  the  use  of  non- 
developmental  items  (NDI);  and 

•  The  shortened  period  of  technological  advances,  and  the  ready  market 
availability  of  commercial  off-the-shelf  (COTS)  components,  change  the 
potential  to  make  performance  trade-offs  and  provides  opportunities  to 
achieve  cost  and  schedule  improvements  (DSMC,  1998:2-2). 


1  “Report  of  the  Defense  Science  Board  Task  force  on  Command  and  Control  Systems  Management’'’,  July 
1978,  Office  of  the  Under  Secretary  of  Defense  Research  and  Engineering,  Washington  D.C.  and 
“Command  and  Control  (C2)  Systems  Acquisition  Study  Final  Report",  September  1,  1982,  The  Armed 
Forces  Communications  and  Electronics  Association,  Falls  Church,  Virginia. 
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In  light  of  these  findings,  the  aforementioned  studies  “have  recommend  the  use  of  an  EA 
strategy  to  permit  orderly,  timely,  and  efficient  development  of  effective  defense  systems 
for  the  type  of  environment  in  which  new  defense  acquisitions  will  be  operated  and 
maintained”  (DSMC,  1998:2-2). 

Faced  with  mounting  pressure,  the  DoD  has  responded.  The  most  recent  version 

of  DoDI  5000.2  Operation  of  the  Defense  Acquisition  System  articulates  current  guidance 

on  acquisition  strategy  development.  The  document  states: 

“The  acquisition  strategy  shah  define  not  only  the  approach  to  be  followed 
in  System  Development  and  Demonstration,  but  also  how  the  program  is 
structured  to  achieve  full  capability.  There  are  two  such  approaches, 
evolutionary  and  single  step  to  full  capability.  An  evolutionary  approach 
is  preferred”  (DOD,  2001a:4.7.3.2.3.3.1). 

In  line  with  the  DoD  guidance,  the  services  have  also  adopted  EA  as  the  preferred 
acquisition  approach.  Specifically,  the  Air  Force  has  formalized  an  EA  policy  within  Air 
Force  Instruction  (AFI)  63-123,  EA  for  C2  Systems.  This  AFI  guides  and  directs  the  use 
of  EA  strategy  using  a  spiral  development  process  in  support  of  the  acquisition  of  C2 
systems.  It  is  important  to  note  that  EA  is  not  solely  applicable  to  this  family  of  systems. 
However,  the  approach  is  particularly  useful  when  software  is  a  key  component  of  the 
systems,  and  software  is  required  for  the  system  to  achieve  its  intended  mission  (DOD, 
2001a:4.7.3.2.3.3.1). 

The  AFI  reiterates  the  findings  of  the  previous  studies  and  expands  upon  the  need 
for  a  tailored  EA  approach: 

“Traditional  DoD  acquisition  processes  developed  during  the  cold-war  era 
were  oriented  toward  larger  systems  designed  for  unique  military 
requirements  and  are  not  often  suitable  for  today’s  rapid  technology  changes 
and  continuous  requirement  refinement”  (DAF,  2000a:2). 
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In  short,  EA  addresses  the  volatility  and  risks  associated  with  modem  weapons  system 
development  and  acquisition  efforts.  Potential  sources  of  volatility  and  risk  include: 

•  Uncertainty  about  details  or  maturity  of  requirements, 

•  Continuous  user  input  and  feedback, 

•  Shortened  technology  insertion  life-cycles, 

•  Schedule  urgency, 

•  Budget  and/or  cost  uncertainty, 

•  Technical  maturity,  and 

•  Feedback  from  test,  evaluations,  experiments,  and  exercises 
(DAF,  2000a:2). 

EA  mitigates  volatility  and  risk  by  allowing  an  acquisition  program  to  respond  to 
changing  conditions,  enabling  each  increment  to  accommodate  the  following  three 
activities:  1)  develop  new  capabilities  supporting  the  operational  requirements  and  goals 
of  the  system,  2)  exploit  opportunities  to  insert  new  technologies  that  reduce  cost  of 
ownership  or  accelerate  fielding  of  new  capabilities  resulting  from  experimentation  or 
technology  demonstrations,  and  3)  refine  current  capabilities  based  on  user  feedback, 
testing,  or  experimentation  (DAF,  2000a:3.3). 

The  spiral  development  process  drives  the  capabilities  and  characteristics  of  each 

EA  increment.  A  high-level  definition  of  this  process  is  as  follows: 

“The  spiral  development  model  is  a  risk-driven  process  model  that  is  used 
to  guide  multi-stakeholder  concurrent  engineering  of  software-intensive 
systems.  It  has  two  main  distinguishing  features.  One  is  a  cyclic 
approach  for  incrementally  growing  a  system’s  degree  of  definition  and 
implementation  while  decreasing  its  degree  of  risk.  The  other  is  a  set  of 
anchor  point  milestones  for  ensuring  stakeholder  commitment  to  feasible 
and  mutually  satisfactory  system  solutions”  (Boehm,  2001:2). 
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Figure  1  presents  a  graphical  depiction  of  the  spiral  development  model.  The  cyclic 
nature  of  the  spiral  model  is  discussed  above.  Rather  than  develop  the  completed  product 
in  one  step,  multiple  cycles  are  performed  with  each  taking  steps  calculated  to  reduce  the 
most  significant  remaining  risks  (Boehm,  2001 :  2).  The  goal  of  spiral  development  is  to 
allow  innovation  in  technology  and  operational  concepts  to  occur  simultaneously  and 
continuously  at  many  levels  and  across  all  functional  lines.  The  result  is  operational 
requirements  evolving  in  parallel  with  system  capabilities  through  “An  iterative  process 
of  idea  generation,  rapid  prototyping,  technology  insertion,  and  operational  testing” 

(DAF,  2000a:4.1.2). 


Figure  1.  Spiral  Development  Process  (Boehm,  2001) 


Prior  to  employing  the  spiral  development  model,  it  is  imperative  to  establish  the 
following  program  attributes: 
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2 

•  A  general  description  of  the  functional  capability  desired  for  the  final  system." 

•  A  concise  statement  of  operational  concepts  for  the  final  system. 

3 

•  A  flexible,  well  planned  overall  open-system  architecture. 

•  A  plan  for  incrementally  achieving  the  desired  total  capability  that  adheres  to 
life-cycle  cost  effectiveness. 

•  Continual  dialogue  and  feedback  among  users,  developers,  supporters,  and 
testers  (DAF,  2000b:  8). 

The  rationale  for  mandating  these  attributes  relates  back  to  the  anchor  point  milestones 
cited  in  the  definition  of  the  spiral  development  model.  Each  anchor  point  milestone  is  a 
specific  artifact  or  condition  that  must  be  attained  at  some  point.  These  milestones  serve 
as  commitment  points  and  progress  checkpoints.  They  impel  the  project  toward 
completion  (Boehm,  2001:  3).  The  aforementioned  programmatic  attributes  fonn  the 
basis  for  the  anchor  point  milestone  reviews. 

The  three  spiral  development  model  anchor  points  are  as  follows: 

•  LCO  (Life  Cycle  Objectives)  -  what  should  the  system  accomplish? 

•  LCA  (Life  Cycle  Architecture)  -  what  is  the  structure  of  the  system? 

•  IOC  (Initial  Operating  Capability)  -  the  first  released  version. 

The  focus  of  the  LCO  review  is  to  ensure  there  is  a  viable  business  case.  The  focus  of  the 
LCA  review  is  to  commit  to  a  single  detailed  definition  of  the  project.  The  project  must 
have  either  eliminated  all  significant  risks  or  put  in  place  an  acceptable  risk  management 
plan.  The  LCA  milestone  is  particularly  important,  as  its  pass/fail  criteria  enable 

2  The  lack  of  specificity  and  detail  in  identifying  the  final  system  capability  distinguishes  EA  from  other 
incremental  strategies  (e.g.,  pre-planned  product  improvement  (P3I))  (AFEA  Guide,  2000:  8). 

3  The  system  architecture  defines  the  partitioning  of  system  components,  flow  of  data,  flow  control,  timing, 
through  put  relationships,  interface  layering,  and  protocol  standards.  A  flexible  architecture  requires  long¬ 
term  tolerance  of  change  (AFEA  Guide,  2000:  8). 
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stakeholders  to  hold  up  projects  attempting  to  proceed  into  evolutionary  or  incremental 
development  without  life-cycle  architecture  (Boehm,  2001 :  8).  The  focus  of  the  IOC 
review  is  to  ensure  the  project  is  ready  for  operations.  Together,  the  anchor  point 
milestones  avoid  “analysis  paralysis”,  unrealistic  expectations,  requirements  creep, 
architectural  drift,  COTS  shortfalls  and  incompatibilities,  unsustainable  architectures, 
traumatic  cutovers,  and  useless  systems  (Boehm,  2001:  8). 
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Figure  2.  Notional  Spiral  Development  Model  (DAF,  2000a) 


Figure  2  presents  AFI  63-123’s  notional  implementation  of  the  spiral 
development  model.  As  prescribed  by  the  model,  the  capabilities  and  characteristics  of 
an  increment  are  defined  in  an  iterative  fashion.  Rather  than  developing  the  entire 
increment  in  one  step,  multiple  cycles  (or  spirals)  are  performed  with  each  cycle  taking 
calculated  steps  to  reduce  the  most  significant  remaining  risks  (Boehm,  2001:2). 
Additionally,  the  increment’s  operational  requirements  evolve  in  parallel  with  system 
capabilities  through  this  process.  The  “Feedback”  nodes  are  consistent  with  the  spiral 
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development  model’s  anchor  point  milestones.  These  decisions  are  comparable  to  LCO, 
LCA,  and  IOC  artifacts,  serving  as  commitment  and  progress  checkpoints.  The 
outcomes/decisions  from  the  feedback  nodes  impel  the  increment  forward. 

Figure  2  portrays  the  spiral  development  model  applied  to  a  single  increment. 
However,  EA  is  characterized  by  the  early  fielding  of  an  initial  (core)  capability, 
enhanced  though  the  delivery  of  additional  increments.  These  additional  increments 
ultimately  contribute  to  a  final  system  capability  (DAF,  2000b:7).  As  previously 
mentioned,  one  of  the  necessary  programmatic  attributes  is  a  cost  effective,  life-cycle 
plan  for  incrementally  achieving  the  desired  total  capability.  Again,  a  major  goal  of  the 
EA  strategy  is  to  deliver  an  operationally  useful  and  supportable  capability  to  the  user 
quicker  than  traditional  strategies.  Therefore,  this  plan  must  focus  on  early  fielding  of 
capability  by  using  mature,  well-understood  technologies  (and  requirements)  for  the  core 
while  saving  higher  risk  activities  for  the  latter  increments  (DAF,  2000b:27).  This  aspect 
of  EA  necessitates  operational  requirements  to  be  time  phased. 

Table  1  presents  a  graphical  depiction  of  time-phased  operational  requirements 
for  a  notional  weapons  system.  The  first  column  contains  the  designation  for  each  of  the 
performance  parameters.  Perfonnance  parameters  are  system  capabilities  or 
characteristics  that  describe  what  the  user  expects  from  the  system  in  order  to  perfonn  the 
mission  and  satisfy  the  mission  requirement.  The  second  column  designates  whether  or 
not  a  performance  parameter  is  a  key  perfonnance  parameter  (KPP).  KPPs  are  those 
capabilities  and  characteristics  considered  most  essential  for  successful  mission 
accomplishment  (DAF,  2000b:25).  The  third  column  describes  performance  parameter 
levels.  A  threshold  is  a  minimum  acceptable  value  for  a  system  capability  or 
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characteristic  that,  in  the  user’s  judgment,  is  necessary  to  provide  the  operational 
capability  that  satisfies  the  mission  need.  An  objective  is  a  value  beyond  the  threshold 
that  could  have  a  measurable  and  beneficial  impact  on  the  system  capability, 
supportability,  or  operational  concept  of  employment  (DAF,  2000b:25).  The  remaining 
columns  specify  the  capabilities  and  characteristics  for  each  of  the  EA  increments.  The 
operational  requirements  are  phased  appropriately  across  the  horizon  of  increments  so  the 
core  provides  an  initial,  operationally-useful  capability  through  the  use  of  readily 
available  technologies.  The  latter  increments  address  other  higher  risk  requirements. 


Table  1.  Time  Phased  Operational  Requirements  (DAF,  2000b) 
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Another  method  for  visualizing  time-phased  requirements  is  through  the  use  of  a 
Venn  diagram  and  a  simplified  Gantt  chart.  The  Venn  diagram  in  Figure  3  illustrates 
how  the  various  increments  combine  to  deliver  the  total  operational  capability  for  a 
notional  weapon  system.  The  Gantt  chart  presents  a  timeline  for  the  execution  and 
delivery  of  each  increment.  It  is  important  to  note,  the  spiral  development  process 
described  by  Figure  2  takes  place  within  each  of  the  rectangular  increments  in  Figure  3. 
The  equations  depict  the  logical  relationship  between  the  operational  requirements  and 
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the  increments.  For  example,  the  core  increment  addresses  the  threshold  level  for  KPP  1, 
the  threshold  level  for  KPP  two,  and  so  on.  The  summation  of  the  individual  increments 
equates  to  the  total  operational  capability  documented  within  the  EA  operational 
requirements  document  (ORD).  Using  the  spiral  development  process  model,  the  ORD 
begins  its  life  as  a  general  description  of  the  functional  capability  desired  for  the  final 
system.  However,  after  successive  spirals  and  increments,  the  ORD  becomes 


increasingly  more  detailed. 


Figure  3.  Graphical  Representation  of  Time  Phased  Requirements  (DAF,  2000b) 


While  there  are  more  aspects  to  the  implementation  of  an  EA  strategy  (e.g., 
contracting  considerations,  operational  testing,  etc.),  the  previous  discussion  provides  the 
level  of  detail  needed  for  the  scope  and  direction  of  this  research.  It  is  now  necessary  to 
review  the  methods  available  to  implement  CAIV  analysis. 
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Implementing  Cost  as  an  Independent  Variable  Analysis 


As  defined  in  the  previous  chapter,  CAIV  is  a  DoD  strategy  that  makes  total  life- 

cycle  cost  as  projected  within  the  acquisition  environment  a  key  driver  of  system 

requirements,  performance  characteristics,  and  schedules  (Rush,  1997:162).  The  Defense 

Acquisition  Deskbook  supplies  a  broader  description: 

“CAIV  is  a  strategy  that  entails  setting  aggressive,  yet  realistic  cost 
objectives  when  defining  operational  requirements  and  acquiring  defense 
systems  and  managing  achievement  of  these  objectives.  Cost  objectives 
must  balance  mission  needs  with  projected  out-year  resources,  taking  into 
account  existing  technology,  maturation  of  new  technologies  and 
anticipated  process  improvements  in  both  DoD  and  industry”  (Kaminski, 

1995:3). 

DoD  Document  5000.2-R,  Mandatory  Procedures  for  Major  Defense  Acquisition 
Programs  (MDAPS)  and  Major  Automated  Information  Systems  (MAIS)  Acquisition 
Programs  has  cemented  this  way  of  thinking  within  DoD  acquisition  policy.  Per  this 
document,  “The  user  shall  treat  cost  as  a  military  requirement.  The  acquisition 
community,  including  technology  and  logistics,  and  the  requirements  community  shall 
use  the  CAIV  process  to  develop  total  ownership  cost  (TOC),  schedule,  and  performance 
thresholds  and  objectives”  (DOD,  200 lc:C  1.3.1). 

RAND  cites  CAIV  as  being,  “Probably  the  single  most  important  element  for 
carrying  out  the  transfonnation  to  commercial-like  weapon  system  R&D  approach” 
(Lorell  and  Grasser,  2001 :32).  In  a  multi-year  study,  RAND  evaluated  the  AR  cost 
saving  estimates  for  eleven  weapon  system  programs  (to  include  the  Joint  Direct  Attack 
Munition  (JDAM),  Joint  Air-to-Surface  Standoff  Missile  (JASSM),  and  more).  Pursuant 
to  this  study,  researchers  looked  at  the  overall  impact  of  CAIV  upon  weapon  system 
costs.  According  to  RAND,  the  data  suggest  that  R&D  savings  in  the  range  of  15  to  35 
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percent  may  be  possible  in  certain  types  of  programs  that  are  structured  in  a  commercial¬ 
like  manner  in  accordance  with  CAIV  (Lorell  and  Grasser,  2001 : 1 19).  However,  the 
researchers  qualify  these  results  by  stating,  “The  AR  (study)  pilot  programs  are  relatively 
small  and  are  characterized  by  low  technological  risk,  commercial  derivative  items,  and 
large  production  runs.  Thus,  the  scale  of  potential  cost  benefits  for  a  large,  complex 
weapon  system  that  employs  high-risk,  cutting-edge  technology  remains  uncertain” 
(Lorell  and  Grasser,  2001:120). 

As  mentioned  previously,  the  commercial  analogue  to  CAIV  is  target  costing, 
also  referred  to  as  “must  cost.”  Under  a  “must  cost”  approach,  a  commercial  developer 
first  conducts  market  research  to  determine  potential  customer  requirements  and  price 
estimates.  Using  these  data,  the  developer  sets  price  and  profit  targets  for  the  finished 
product.  The  difference  between  these  two  values  yields  the  target  or  “must  cost.”  The 
target  cost  is  then  distributed  to  the  various  product  subsystems.  The  subsystem  targets 
costs  are  further  decomposed  and  passed  along  the  design  and  supply  chains. 

In  a  survey  of  aerospace  firms  that  do  business  in  the  commercial  sector  (to 
include  Boeing,  Lockheed  Martin,  Northrop  Grumman,  et  ah),  RAND  researchers  noted 
the  following: 

•  The  “must  cost”  approach  delivers  safe,  reliable  aircraft  to  the 
airlines  at  extremely  competitive  prices.  However,  budget-induced 
design  conservatism  may  also  reduce  both  the  size  and  scope  of 
purely  perfonnance  related  technological  innovations  in  the 
commercial  aircraft  industry. 

•  Under  “must  cost”,  commercial  carriers  are  generally  not  willing  to 
pay  for  technology  innovations  that  improve  the  performance  of 
aircraft  equipment  unless  they  believe  those  improvements  will 
contribute  to  their  immediate  bottom-line  profitability. 
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•  With  the  move  toward  incrementalism  introduced  by  “must  cost”, 
performance-centered  innovations  may  be  less  likely  to  appear 
(Lorellet  al„  2000:110-11). 

In  the  context  of  adopting  a  commercial-like  approach  to  weapon  system 
acquisition,  the  results  from  this  survey  beg  an  important  question.  Can  system  cost  be 
reduced  without  sacrificing  performance?  RAND  believes  that  adopting  a  commercial- 
like  acquisition  strategy  will  prove  beneficial  to  the  DoD.  The  researchers  found  that 
binding  cost  constraints  introduced  by  “must  cost”  have  shifted  the  focus  of  commercial 
aerospace  manufacturers  from  performance  to  cost.  This  has  not  resulted  in  airliners  with 
poor  performance  characteristics  (in  some  cases  there  have  been  notable  improvements) 
(Lorell  et  al.,  2000: 135).  However,  when  adopting  a  “must  cost”  approach  (i.e.,  CAIV), 
the  DoD  must  demand  careful  program  management  to  sustain  technical  innovation  in  the 
desired  areas  (Lorell  et  al.,  2000:135). 

The  “careful  program  management”  cited  in  the  RAND  study  is  manifested  by 
disciplined  requirements-cost-performance  trades-offs;  the  essence  of  CAIV 
implementation  (Rush  1997:  163).  According  to  DoD  5000. 2-R,  “The  best  time  to 
reduce  TOC  and  program  schedule  is  early  in  the  acquisition  process.  Continuous  cost  / 
schedule  /  performance  trade-off  analysis  shall  accomplish  cost  and  schedule  reductions” 
(DOD,  2001c:C1.3.3.1).  The  logic  behind  CAIV’s  emphasis  on  trade-offs  is  twofold. 
First,  system  costs  are  constrained.  While  some  programs  do  obtain  additional  funding 
when  needed,  such  funding  is  often  at  the  expense  of  other  programs  or  future 
modernization.  Second,  understanding  “trade  space”  is  the  foundation  for  smart  decision 
making.  Trade  space  is  the  range  of  alternatives  available  to  decision  makers.  It  is  four¬ 
dimensional;  comprising  performance,  cost  (i.e.,  TOC),  schedule,  and  risk  impacts  (Kaye 
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et  al.,  2000:354).  The  trade-off  process  is  more  effective  if  it  can  be  accomplished  earlier 
in  the  acquisition  life-cycle  of  a  system.  A  large  percentage  of  cost  is  determined  by  a 
small  percentage  of  the  decisions.  These  critical,  cost-driving  design  decisions  are  made 
early  in  the  concept  selection  and  design  process  (Rush,  1997:165). 

According  to  Kaye  et  al.,  “Clear  identification  and  use  of  viable  trade  space,  or 
the  range  of  alternatives,  with  full  knowledge  of  real  and  potential  impacts  is  essential  for 
making  the  right  decisions  to  meet  user  needs  while  controlling  cost”  (Kaye  et  al., 
2000:355).  Trade  space  is  commonly  defined  for  alternatives  in  terms  of  performance, 
cost,  and  schedule  impacts  that  each  alternative  presents  (Kaye  et  al.,  2000:355).  Risk 
must  also  be  addressed.  Risk  drives  many  critical  decisions  and  is  a  fourth  dimension  in 
the  trade  space.  Additionally,  risk  “discounts”  the  anticipated  performance,  cost,  and 
schedule  options;  it  restricts  trade  space  (Kaye  et  al.,  2000:355). 

Figure  4  depicts  the  cost-performance  trade  space  of  a  KPP  for  a  notional  weapon 
system.  The  KPP  is  characterized  by  threshold  and  objective  levels,  found  on  the 
performance-axis.  The  KPP’s  cost  is  bounded  within  a  predetermined  life-cycle  cost 
target.  The  shaded  region  includes  all  feasible  solutions.  The  “solution  set”  line  equates 
the  optimum  cost-performance  combinations.  Feasible  solutions  not  found  on  the 
solution  set  line  are  sub-optimal,  meaning  more  performance  for  equal  cost  or  equal 
performance  at  less  cost  is  possible  (Kaye  et  al.,  2000:  356).  The  “risk  reserve”  line 
constrains  the  trade  space  and  limits  the  region  of  feasible  solutions.  Trade  spaces,  like 
the  one  depicted  in  Figure  4,  exist  for  all  system  performance  parameters  (both  key  and 
non-key). 
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Figure  4.  Cost  /  Performance  Trade  Space  (Kaye  et  al.,  2000) 


While  trade-offs  are  central  to  CAIV  implementation,  risk  management  is  integral 
as  well  (Kaye  et  al.,  2000:361).  Risk  management’s  role  recognizes  that  a  program 
cannot  afford  to  avoid  all  risk,  but  rather  must  manage  critical  risks  (Kaye  et  al., 
2000:356).  Because  risk  influences  the  available  trade  space,  risk  reduction  measures 
must  be  addressed  when  performing  cost-performance  trade-offs. 

Identification  of  the  trade  space  is  followed  by  rigorous  and  fonnal  cost/benefit 
trade-off  analyses;  beginning  at  initial  concept  development  and  continuing  into 
production  and  sustainment.  One  of  the  primary  goals  of  this  analysis  is  to  identify  the 
“knee  of  the  curve”  after  which  each  marginal  increase  in  capability  or  performance 
becomes  increasingly  expensive  (Lorell  and  Graser,  2001:34).  This  analysis  is  necessary 
so  that  the  user  understands  the  cost  of  increasing  perfonnance  in  any  given  area  and 
recognizes  at  what  point  the  phenomenon  of  diminishing  marginal  returns  comes  into 
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play.  Thus,  the  user  community  can  make  informed  judgments  regarding  the  priority  of 
performance  requirements  and  the  allocation  of  resources  (Lorell  and  Graser,  2001:  36  ). 

CAIV  implementation  relies  upon  the  use  of  capability-based  requirements  (Kaye 
et  al.,  2000:357).  Instead  of  specifying  how  to  build  a  system  and  how  to  allocate 
subsystems,  the  user  must  instead  state  what  the  system  needs  to  bring  to  the  fight.  This 
approach  to  system  definition  increases  flexibility  and  further  aids  the  development  team 
in  delivering  the  “best-value”  system  that  meets  user  operational  requirements  (Kaye  et 
al.,  2000:356).  The  user  must  then  carefully  prioritize  the  mission  performance  needs 
and  capability-based  requirements.  Prioritization  is  critical  to  facilitate  intelligent  trade¬ 
offs  between  cost  and  capability.  A  key  objective  of  prioritization  is  to  avoid  “over 
designing”  or  “gold-plating”  weapon  systems  with  higher  performance  and  more 
extensive  capabilities  that  are  not  truly  necessary  to  perform  the  mission  (Lorell  and 
Graser,  2001 :34).  Thus,  prioritization  helps  to  exclude  nonessential  requirements  while 
helping  the  development  team  maximize  use  of  the  trade  space  by  focusing  on 
characteristics  contributing  most  to  mission  accomplishment  (Kaye  et  al.,  2000:356). 

Beyond  simple  prioritization,  it  is  essential  to  understand  the  explicit  and  implicit 
relations  between  the  individual  capabilities-based  requirements  (or  performance 
parameters)  (Wollover,  1997:317).  A  means  is  required  to  systematically  organize  all  of 
these  variables  and  their  interrelationship  (Wollover,  1997:317).  Quality  function 
deployment  (QFD)  is  a  well-established  procedure  used  to  organize  and  translate  user 
requirements.  QFD  has  been  used  extensively,  across  many  industrial  sectors,  to 
translate  and  map  user  needs  into  objective  system  outcomes  (Wollover,  1997:  318).  The 
literature  indicates  that  QFD  is  the  most  widespread  implementation  methodology  for 
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total  quality  management  (TQM)  (Sage,  1992:  222).  QFD  is  a  process  tool  that  enhances 
a  development  team’s  ability  to  manage  key  elements  of  the  system  engineering  process 
(Wollover,  1997:  318). 

Through  a  series  of  interdependent  matrices,  QFD  accommodates  vaguely  stated 
customer  specifications.  These  matrices  allocate  and  map  requirements  into  specific 
design  strategies,  development  processes,  and  system  characteristics  (Wollover,  1997: 
318).  For  each  element  of  the  system  design,  technical  performance  measures  (TPMs) 
are  addressed  and  threshold/objective  values  assigned.  Using  an  iterative  process,  these 
assignments  set  the  minimum  levels  of  achievement  required  to  satisfy  the  user’s  overall 
requirements. 

The  literature  reveals  that  QFD  was  developed  in  the  late  1960s  by  Shigeru 
Mizano  of  the  Tokyo  Institute  of  Technology  (Menon  et  ah,  1994:  94).  Around  this  time, 
Mitsubishi  Heavy  industries  began  to  use  QFD  on  supertanker  projects.  These  projects 
were  characterized  by  having  sophisticated  propulsion,  maneuvering,  and  balance 
control,  challenging  design  and  manufacturing  logistical  requirements  (Guinta  et  ah, 

1993:  1).  Toyota  then  adopted  the  Kobe  shipyard  QFD  strategy,  modified  its 
methodology,  and  experienced  40  percent  reductions  in  new  model  development  costs 
and  50  percent  reductions  in  development  time  (Menon  et  ah,  1994:  94).  U.S.  firms  such 
as  Ford,  Ernst  and  Young,  Texas  Instruments,  General  Motors,  ITT,  and  IBM  have  also 
embraced  QFD  strategies.  Research  reveals  that  various  domestic  manufacturing 
companies  using  QFD  have  experienced  50  percent  cost  reductions  and  33  percent 
project  time  reductions  (Guinta  et  ah,  1993:  8).  The  DoD  Joint  Strike  Fighter  (JSF) 
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program  has  adopted  QFD  techniques  and  has  been  recognized  for  its  aggressive 
implementation  to  better  analyze  weapons  system  requirements  (Wollover,  1997:320). 

The  DoD’s  emphasis  upon  integrated  product  and  process  development  (IPPD) 
and  the  integrated  product  team  (IPT)  structure  enhances  the  applicability  of  QFD  to 
CAIV  implementation  (Wollover,  1997:  320).  Precedent  dictates  that  cost/performance 
IPTs  (CPIPTs)  oversee  the  execution  of  CAIV  initiatives  within  DoD  programs.  QFD 
provides  the  means  to  trace  cost  objectives  as  they  are  decomposed  from  the  system  to 
the  sub-systems  level.  The  CPIPT  may  then  use  the  QFD  products  to  recommend 
engineering  and  design  changes  to  the  program  manager  so  that  CAIV  objectives  are  met 
(Wollover,  1997:320). 

QFD  assists  CAIV  implementation  in  several  ways.  Most  directly,  QFD 
comprehensively  displays  relationships  between  various  cost  variables  (i.e.,  cost  drivers). 
This  aspect  leads  to  more  structured  analyses  and  more  intelligent  prioritization  schemes. 
The  addition  of  technical  performance  measure  (TPM)  difficulty  as  a  measure  of  risk 
further  improves  the  quality  of  information  available  to  assist  in  trade-off  decisions. 
Finally,  the  multiattribute  structure  of  the  QFD  matrix  captures  and  interrelates  the  data 
necessary  to  design  and  evaluate  multiattribute  optimization  problems  (Wollover, 
1997:330). 

The  topic  of  system  design  optimization  through  QFD  is  addressed  by  Thurston 
and  Essington,  Thurston  and  Locascio,  and  Fung  et  al..  Thurston  and  Essington  explain 
how  the  weighted  average  method  (i.e.  prioritization)  commonly  used  to  optimize  designs 
has  limitations  because  it  does  not  accurately  reflect  the  nonlinear  value  imparted  by 
performance  parameters  (Thurston  and  Essington,  1993:48).  Instead,  the  authors  employ 
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a  utility  theory-based  model  that  incorporates  user  willingness  to  make  trade-offs 
between  performance  parameters.  Thurston  and  Locascio  emphasize  the  importance  of 
considering  economic  or  non-technical  factors  when  evaluating  product  designs 
(Thurston  and  Locascio,  1994:41).  The  authors  demonstrate  an  analysis  technique  that 
allows  designers  to  treat  economic  factors  with  the  same  respect  they  traditionally  give  to 
technical  factors  (Thurston  and  Locascio,  1994:41).  Fung  et  al.  integrate  imprecision  and 
uncertainty  with  a  QFD-based  multiattribute  optimization  problem  formulation.  The 
ultimate  goal  of  the  model  proposed  by  Fung  et  al.  is  to  help  decision  makers  deploy 
design  resources  in  a  manner  that  improves  overall  customer  satisfaction  (Fung  et  al., 
2002:585). 

More  directly  related  to  CAIV,  research  by  Luman  presents  an  implementation 
process  to  support  complex  systems  requirements  allocation  as  a  function  of  cost. 

Luman’ s  research  attempts  to  answer  the  question,  “From  the  systems  of  systems 
performance  perspective,  where  are  the  limited  resources  best  applied”  (Luman,  1999:8)? 
Through  this  process,  Luman  covers  a  broader  category  of  CAIV  implementation  by 
addressing  “systems  of  systems”  issues.  Systems  of  systems  are  generally  viewed  as 
having  the  following  characteristics: 

•  The  system  is  comprised  by  several  independently  acquired  systems, 
each  under  a  nominal  systems  engineering  process; 

•  Time  phasing  between  each  systems  system’s  development  is  arbitrary 
and  not  contractually  related; 

•  System  couplings  are  neither  totally  dependent  nor  independent,  but 
rather  interdependent; 

•  Individual  systems  are  generally  unifunctional  when  viewed  from  the 
system  of  systems  perspective; 
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•  Optimization  of  each  system  does  not  guarantee  overall  system  of 
systems  optimization;  and 

•  Combined  operation  of  the  systems  constitutes  and  represents 
satisfaction  of  an  overall  mission  or  objective  (Luman,  1999:8). 

From  the  "systems  of  system  perspective,”  Luman’ s  methodology  presents  two 
potential  CAIV  objectives.  The  first  seeks  to  determine  the  optimal  allocation  of 
resources  (developing  new  systems,  modifying  legacy  systems,  inserting  advanced 
technology,  or  implementing  a  combination  of  these  options)  as  a  function  of  total  cost. 
The  second  objective  looks  to  optimize  a  specified  top  level  measure  of  effectiveness 
(MOE)  within  the  bounds  of  the  stated  constraints  (Luman,  1999:8).  It  is  possible  to  pare 
Luman’ s  systems  of  system  CAIV  implementation  methodology  to  address  just  this 
second  objective  in  a  narrower,  discrete  (non-system  of  systems)  system  context. 

Figure  5  is  a  graphic  representation  of  Luman’ s  methodology.  The  process  is 
characterized  by  two  phases.  Phase  I  involves  developing  closed  form  equations  that 
relate  system  design  components  and  parameters  to  system  effectiveness.  In  this  phase,  a 
single  overarching  MOE  for  the  system,  characterizing  mission  success,  is  defined.  This 
top-level  MOE  is  related  (via  equations)  to  multiple  measures  of  performance  (MOPs). 
The  MOPs  correspond  to  system  performance  parameters  (both  key  and  non-key).  Initial 
boundary  conditions  and  constraints  are  then  specified  for  the  system  MOPs  (e.g.,  cost 
targets,  technological  bounds,  force  structure  limitations,  etc.).  Performance  based  cost 
models  (PBCMs)  are  developed  to  calculate  cost  as  a  function  of  the  parameterized 
MOPs.  Phase  II  then  implements  simulation  techniques  to  solve  the  resulting 
constrained,  nonlinear  (stochastic)  performance  problem.  Simulations  are  repeated, 
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gradually  relaxing  the  overall  cost  constraint.  Finally,  sensitivity  analysis  is  performed  to 
understand  the  influence  of  the  non-cost  constraints  (Luman,  1998:6). 
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Figure  5.  System  of  Systems  CAIV  Methodology  (Luman,  1999) 


Luman  cites  the  following  challenges  to  be  wary  of  when  implementing  this 
CAIV  methodology: 

•  Defining  the  overarching  MOE, 

•  Allocation  of  system  components  and  selection  of  trade  space  for 
MOPs, 

•  Adaptation/adoption  of  appropriate  PBCMs, 

•  Application  of  efficient  and  appropriate  optimization  algorithms,  and 

•  Verification  and  validation  of  process  models  (Luman,  1999: 11) 
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While  the  methodology  focuses  on  how  best  to  upgrade  complex  systems  of  systems,  the 
process  can  be  reduced  to  find  the  “best”  range  of  solutions  for  a  particular  system 
subject  to  cost,  operational,  and  technological  constraints,  relative  to  an  overarching 
measure  of  effectiveness. 

Further  work  on  CAIV  implementation  methodology  has  been  conducted  by  the 
Systems  Management  and  Production  Laboratory  (SMAPLAB),  an  applied  research  arm 
of  the  U.S.  Army  Aviation  and  Missile  Command  (AMCOM),  NASA,  and  the  University 
of  Alabama.  The  SMAPLAB  CAIV  model  is  an  electronic  tool  designed  to  support 
program  management  office  (PMO)  level  IPTs  trade-off  analyses  among  cost, 
performance,  and  schedule  elements.  Utilizing  a  QFD-like  approach,  the  SMAPLAB 
tool  allows  users  to  enter  perfonnance  requirements  and  design  characteristics,  their 
correlations,  and  priority  rankings.  Using  this  data,  the  model  outputs  the  critical 
relationships  between  pairs  of  performance  requirements  and  design  characteristic.  The 
model  also  identifies  performance  requirements  that  are  most  sensitive  to  changes  in 
design  characteristics.  Currently,  the  model  does  not  integrate  cost,  perfonnance,  and 
schedule  information.  Additionally,  the  model  does  not  provide  values  for  the  magnitude 
of  trade-off  impacts  (Mullins,  1998:7-9). 

Tecolote  Research,  Inc.  has  integrated  a  “first  order”  CAIV  capability  within  the 
Automated  Cost  Estimating  Integrated  Tools  (ACEIT)  software  suite  (version  5.x).  With 
this  capability,  one  can  set  cost  targets  or  time-phased  budgets  and  obtain  insight  into 
how  the  driver  within  the  cost  estimating  methodology  is  affected.  An  optimization 
algorithm  generates  a  solution  that  satisfies  the  constraints  specified  for  system  cost  and  a 
single  cost  driver,  or  “free  variable.”  The  marketing  literature  for  this  tool  states,  “This 
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function  is  not  meant  to  solve  all  of  an  organization’s  CAIV  issues,  but  rather  provide  a 
means  to  gauge  the  impacts  on  cost  estimating  methodology  drivers  and  provide  direction 
for  more  thorough  investigation”  (Tecolote,  2002).  Currently,  the  tool  does  not  integrate 
requirements  prioritization,  an  area  of  primary  concern  in  CAIV  analysis.  Additionally, 
the  solver  algorithm  employed  by  the  tool  is  rather  limited  and  does  not  allow  the  user  to 
vary  more  than  one  decision  variable.  This  limitation  of  the  ACEIT  approach  thus 
hinders  a  holistic  view  when  attempting  to  conduct  CAIV  trade-offs. 

Utility  Theory 

Earlier,  the  topic  of  utility  theory  was  mentioned  when  describing  techniques  for 
system  design  optimization  with  QFD.  Because  this  concept  plays  a  pivotal  role  in  the 
formulation  of  the  CAIV/EA  model,  it  is  important  to  present  a  brief  survey  of  utility 
theory  and  its  application  to  the  overarching  practice  of  decision  analysis. 

As  described  by  Ragsdale  (2001),  the  goal  of  decision  analysis  is  to  help 
individuals  make  good  decisions.  Although  all  decision  problems  are  somewhat 
different,  they  share  certain  characteristics  (Ragsdale,  2001:714).  The  following  is  a 
brief  (non-exhaustive)  list  of  the  general  characteristics  of  a  decision  problem: 

•  There  exists  at  least  two  alternatives  for  addressing  or  solving  the  problem; 

•  An  alternative  is  a  course  of  action  intended  to  solve  the  problem; 

•  Alternatives  are  evaluated  on  the  basis  of  the  value  they  add  to  one  or  more  of 
the  decision  criteria;  and 

•  The  criteria  represent  various  factors  that  are  important  to  the  decision  maker 
(Ragsdale,  2001:714-15). 

Often  a  decision  maker  is  a  faced  with  multiple  criteria  when  evaluating  a 
decision  problem.  Many  times,  these  criteria  compete  or  conflict  with  one  another. 
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Utility  theory  presents  one  approach  to  assessing  trade-offs  between  multiple  criteria. 
Additionally,  utility  theory  provides  a  means  to  incorporate  a  decision  maker’s  attitude 
and  preference  toward  risk  and  return  in  the  decision  analysis  process  so  that  the  most 
desirable  decision  alternative  is  identified.  Utility  theory  assumes  that  every  decision 
maker  uses  a  utility  function  that  translates  each  of  the  possible  alternatives  in  a  decision 
problem  into  a  non-monetary  measure  called  utility.  Utility  represents  the  total  worth, 
value,  or  desirability  of  the  outcome  of  a  decision  alternative  to  the  decision  maker. 

Often  utilities  are  represented  on  a  scale  from  zero  (0)  to  one  (1),  where  0  indicates  that 
the  outcome  of  the  alternative  has  no  value  to  the  decision  maker  and  1  represents  perfect 
or  superior  value  (Ragsdale,  2001:757). 


Utility  Functions 


1.20  , 


Payoff 


RiskAverse-  ■■  » Risk  Neutral  Risk  Seeking 


Figure  6.  Utility  Functions  (Ragsdale,  2001) 

Figure  6  illustrates  three  different  decision  maker  attitudes  toward  risk. 
According  to  Ragsdale: 

“A  ‘risk  averse’  decision  maker  assigns  the  largest  relative  utility  to  any  payoff 
but  has  a  diminishing  marginal  utility  for  increased  payoffs  (that  is,  every 
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additional  increment  of  payoff  results  in  smaller  increases  in  utility.  A  ‘risk 
seeking’  decision  maker  assigns  the  smallest  utility  to  any  payoff  but  has  an 
increasing  marginal  utility  for  increased  payoffs  (that  is,  every  additional 
increment  of  payoff  results  in  larger  increase  in  utility.  The  ‘risk  neutral’  decision 
maker  falls  in  between  these  two  extremes  and  has  a  constant  marginal  utility  for 
increased  payoffs  (that  is,  every  additional  dollar  in  payoff  results  in  the  same 
amount  of  increase  in  utility)”  (Ragsdale,  2001:757-58). 

Applying  utility  functions  to  the  criteria  composing  a  multiattribute  decision 

problem  allows  a  decision  maker  to  execute  rigorous,  quantitative  trade-offs.  When  there 

are  multiple,  competing  criteria,  it  is  often  challenging  to  reduce  a  decision  to  a  single 

dimension.  Fortunately,  individual  utility  functions  for  the  various  decision  criteria  can 

be  synthesized  into  an  overall  utility  function  that  measures  the  decision  maker’s  overall 

satisfaction  for  a  given  alternative.  This  approach  to  using  utility  theory  to  address 

multiattribute  decision  problems  is  fully  developed  in  the  next  chapter. 
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III.  Methodology 


Overview 

This  chapter  describes  and  substantiates  the  methodology  used  to  integrate  CAIV 
analysis  within  an  EA  strategy.  First,  the  core  CAIV  model  is  formulated.  Following 
this  formulation,  the  model  is  expanded  to  incorporate  the  features  of  EA  (e.g.,  time¬ 
phasing  and  technical  risk  mitigation).  Finally,  potential  techniques  for  model  validation 
are  discussed. 

Core  CAIV  Model  Formulation 

Based  upon  information  presented  in  the  previous  chapter,  the  essence  of  CAIV 
implementation  is  embodied  by  disciplined  cost-performance  trade-off  analysis  (Rush 
1997:  163).  It  is  possible  to  model  these  trade-offs  through  the  use  of  multiattribute 
design  evaluation,  incorporating  economic  factors  as  measures  of  performance.  The 
challenge  in  perfonning  this  type  of  evaluation  lies  in  developing  an  objective  function 
that  clearly  and  accurately  integrates  the  various  measures  of  performance  associated 
with  the  design. 

Luman  uses  an  overarching  system  measure  of  effectiveness  (MOE)  that  is 
mathematically  linked  to  individual  systems’  measures  of  performance  (MOP)  as  an 
objective  function  in  his  methodology  (Luman,  1998:6).  According  to  Thurston  and 
Essington,  “Recent  efforts  to  include  manufacturing  cost  considerations  in  the  design 
process  incorporate  a  step  in  which  design  alternatives  are  compared  on  the  basis  of  their 
performance  in  several  attributes.  The  most  common  method  used  in  this  type  of 
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multiple  attribute  evaluation  of  a  design  is  some  form  of  weighted  average”  (Thurston 
and  Essington,  1993:49). 

The  weighted  average  approach  employs  the  following  functional  form: 

n 

^(x)=Xw'’x'  (i) 

i= 1 

In  this  formulation,  T(x)  is  the  total  worth  of  an  alternative  characterized  by  attribute 
vector  x  =  (x\,  ...  ,  x„);  x,  is  the  level  of  the  perfonnance  attribute  i;  i  equals  the  1,2,  . . .,  n 
attributes;  and  vv;  is  the  weighting  factor  (Thurston  and  Essington,  1993:49). 

According  to  Thurston  and  Essington,  this  approach  has  two  limitations.  First,  it 
assumes  a  linear  relationship  between  the  level  of  an  attribute  x,  and  its  subsequent  worth 
or  value  to  the  decision  maker.  There  are  many  instances  where  this  relationship  is  not 
linear,  because  decision  makers  do  not  attach  the  same  value  to  each  unit  of  benefit  they 
receive  or  expense  they  pay  (Thurston  and  Essington,  1993:49).  Figure  7  illustrates  a 
notional  non-linear  relationship  between  value  and  perfonnance. 


Notional  Performance  Attribute 


Figure  7.  Notional  Non-Linear  Relationship  (Thurston  and  Essington,  1993) 
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Second,  the  approach  does  not  accurately  capture  the  trade-offs  decision  makers  are 
willing  to  make  between  attributes.  This  is  the  result  of  weighting  factors  being  assigned 
values  based  upon  an  ad-hoc  assessment  of  relative  importance  of  one  attribute  to  another 
rather  than  the  decision  maker’s  willingness  to  make  trade-offs  between  attributes 
(Thurston  and  Essington,  1993:49). 

For  the  reasons  listed  above,  Thurston  and  Essington  assert  that  utility  analysis  is 
superior  to  conventional  weighted  average  methods  for  multiattribute  design  evaluation. 
In  general,  this  approach  disaggregates  a  complex  and  difficult  decision-making  problem 
into  separate  components.  Next,  the  decision  maker’s  statements  of  preference  for  each 
component  are  collected.  Finally,  the  components  are  reassembled  to  provide 
overarching  guidance  (Thurston  and  Essington,  1993:50). 

The  general  form  of  the  multiplicative  multiattribute  utility  analysis  objective 
function  is  listed  below: 


U(x) 


K 


i=l 


(2) 


In  this  formulation,  U(x)  is  the  overall  utility  of  an  alternative  characterized  by 
performance  attribute  vector  x  =  {x\,  ...  ,  xn);  x,  is  the  level  of  the  performance  attribute  i; 
Ufa )  is  the  single  performance  attribute  utility  function  for  attribute  i;  i  equals  the 
1,2 ,...,«  attributes;  k,  is  the  single  performance  attribute  scaling  constant;  and  K  is  the 
nonnalizing  constant  (Thurston  and  Essington,  1993:50).  Values  for  the  single 
performance  attribute  utility  functions  range  from  zero  to  one.  When  all  performance 
attributes  are  at  their  best,  the  overall  utility  equals  one.  Conversely,  when  all  of  the 
performance  attributes  are  at  their  worst,  the  overall  utility  is  set  equal  to  zero. 
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Thurston  and  Essington  emphasize  the  point  that  the  scaling  constants,  ki,  are  not 
arbitrarily  assigned  weighting  factors,  nor  do  they  imply  relative  importance  of  attributes. 
Instead,  kt,  imply  the  decision  maker’s  willingness  to  make  trade-offs  between 
performance  attributes  (Thurston  and  Essington,  1993:50).  The  normalizing  constant,  K, 
is  derived  from  the  following: 

l  +  K  =  f\{l  +  K-lc)  (3) 

;= 1 

The  single  performance  attribute  scaling  constants,  vector  k,  are  derived  from  the  overall 
utility  function  (Equation  (2))  when  performance  attribute  level  x,  is  at  its  best  and  all 
other  attributes  are  at  their  worst.  Ultimately,  these  scaling  constants  represent  the  user’s 
willingness  to  improve  in  one  performance  attribute  while  incurring  changes  in 
competing  attributes  (Thurston  and  Locascio,  1994:50) 

From  a  CAIV  perspective,  the  perfonnance  attribute  vector  x  =  {x\,  ...  ,  x„)  is 
analogous  to  the  weapons  system  performance  parameters  specified  in  the  ORD. 

Working  in  concert  with  the  user,  it  is  possible  to  develop  single  performance  attribute 
utility  functions  for  each  of  the  performance  parameters.  These  single  attribute  utility 
(SAU)  functions  are  based  upon  the  specified  threshold  and  objective  levels  of 
performance  for  the  parameters.  Additionally,  each  SAU  function  represents  the  value 
the  user  places  upon  marginal  improvements  in  performance  for  the  respective  parameter. 
Using  the  technique  described  by  Thurston  and  Essington,  it  is  possible  to  assign  values 
to  the  scaling  constants,  vector  k,  by  evaluating  the  overall  utility  function  when 
performance  parameter  x,  is  at  its  best  and  all  other  performance  parameters  are  at  their 
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worst.  Having  populated  the  scaling  constant  vector,  it  is  then  possible  to  detennine  the 
nonnalizing  constant  K  through  the  use  of  Equation  (3). 

Preparing  the  overall  utility  objective  function  is  an  important  step  in  formulating 
the  CAIV  analysis  problem.  However,  as  it  stands,  the  formulation  is  incomplete.  As 
might  be  inferred  from  the  previous  paragraph,  the  decision  variables  used  in  Equation 
(2)  are  defined  in  terms  of  performance.  Consequently,  this  form  limits  the  incorporation 
of  economic  considerations  into  the  utility  analysis.  The  primary  reasoning  for  this 
limitation  is  that  it  is  difficult  to  assess  the  cost  of  an  alternative  based  solely  on  the 
levels  of  the  performance  parameters.  Instead,  it  is  necessary  to  associate  the 
performance  attributes  with  weapon  system  design  attributes.  Having  developed  the 
design  attribute  vector,  z  =  (z.\,  ....  zm);  where  zj  is  the  level  of  design  attribute  /;  and  m 
equals  the  1,  2,  ...  ,  m  attributes;  it  is  then  possible  to  employ  conventional  cost 
estimating  methodologies  to  determine  the  economic  cost  of  an  alternative. 

Thurston  and  Locascio  describe  how  the  overall  utility  function  can  be  modified 
to  incorporate  design  attributes.  By  determining  the  relationship  between  the  design 
attribute  vector  (z)  that  directly  controls  the  performance  attribute  vector  (x),  one  can  then 
define  the  performance  attribute  function  as  the  following: 

X  =  g(z)  (4) 

The  definition  of  the  performance  attribute  vector  in  terms  of  design  attributes  results  in  a 
modification  to  the  overall  utility  function: 

U(x)  =  U{g(z))  (5) 

Thus,  the  overall  utility  of  an  alternative  is  now  defined  by  the  levels  of  the  design 
attribute  vector  (Thurston  and  Locascio,  1994:64). 
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Defining  an  alternative  in  terms  of  its  design  attributes  offers  several  advantages 
when  performing  CAIV  analysis.  First,  system  developers  make  direct  decisions  on 
design  attributes.  Attained  performance  levels  are  a  result  of  these  design  decisions.  For 
example,  a  mechanical  engineer  employs  a  certain  design  geometry  to  meet  a  given 
strength  (i.e.,  perfonnance)  requirement;  not  the  other  way  around  (Thurston  and 
Locascio,  1994:  64).  The  second  advantage  lies  in  the  opportunity  to  expand  the  trade 
space.  While  the  number  of  performance  attributes  is  fixed,  the  number  of  design 
attributes  is  theoretically  infinite.  The  developer  is  limited  only  by  his  imagination  and 
the  realm  of  the  possible  when  synthesizing  the  design  attribute  vector  for  Equation  (4). 
The  final  advantage  has  already  been  cited.  Current  cost  estimating  models  are  calibrated 
to  derive  cost  as  a  function  of  design  attributes,  not  perfonnance. 

The  remaining  challenge  in  formulating  the  core  CAIV  model  is  generating  the 
function  specified  by  Equation  (4).  This  function  derives  the  performance  attribute  (PA) 
vector  from  the  design  attribute  (DA)  vector.  As  previously  described,  QFD  presents  a 
rigorous  technique  for  tracing  customer  requirements  (i.e.,  PA)  to  design  alternatives 
(i.e.,  DA).  Fung  et  al.  describe  the  QFD  matrix  which  expresses  the  relationship 
between  PA  and  DA.  The  relationship  matrix,  with  elements  Ry,  indicates  the  strength  of 
the  relationship  between  the  /th  performance  and  the yth  design  attributes.  Ry  is 
quantified  on  a  specified  scale  (Fung  et  al.,  2002:587).  If  increasing  the  level  of  zj 
improves  x„  then  a  positive  relationship  exists.  Conversely,  if  increasing  the  level  of  z7 
degrades  xh  a  negative  relationship  is  present.  On  a  -3  to  3  scale,  a  strong  positive 
relationship  between  x,  and  zj  is  given  a  value  of  positive  three.  A  strong  negative 
relationship  is  denoted  by  negative  three.  A  negligible  relationship  is  indicated  by  no 
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entry  in  the  matrix  element  (equals  zero  (0)).  Table  2  depicts  a  notional  QFD  relationship 
matrix: 


Table  2.  Notional  QFD  Relationship  Matrix 


Design  Attribute 

Performance 

Attribute 

1 

2 

3 

4 

5 

6 

1 

-3 

2 

2 

1 

2 

3 

3 

1 

1 

1 

3 

2 

-2 

-2 

3 

3 

-1 

4 

3 

3 

5 

2 

3 

2 

1 

-1 

Again,  the  system  developer  only  controls  the  level  of  attainment  for  the  design 


attributes,  z;.  Therefore,  the  decision  variable  reduces  to  z;.  To  proceed,  it  is  necessary  to 


nonnalize  the  DA  level  of  attainment  by  comparing  z,  to  the  maximum  estimated  level  of 


attainment  for  that  attribute,  zjmax.  Any  attainment  beyond  zjmax  is  assumed  to  generate  no 


value  to  the  design.  Thus,  zj  is  constrained  by  zjmax.  The  ratio  of  zj  to  zjmax  returns  the 


nonnalized  level  of  attainment  for  the  design  attribute,  zj 


Rj  also  requires  nonnalization.  Normalization  is  accomplished  by  dividing  R,j  by 


the  sum  of  the  absolute  values  of  the  matrix  elements  for  performance  attribute  i.  This 


ratio  produces  the  nonnalized  relationship  index,  R  (/  °.  After  normalizing  both  R,,  and  z,- 


it  is  then  possible  to  calculate  the  corresponding  values  for  the  normalized  PA,  x ,  0 .  The 


following  equation  describes  how  the  normalized  PA  is  derived  from  the  normalized  DA 


and  relationship  index: 


m 

6  _  V  T)6  9 

Xi  ZjKij'Zj  (6) 

7=1 


After  calculating  the  normalized  level  of  attainment  for  a  given  PA,  it  is  necessary 


to  determine  how  the  relative  zero  to  one  range  translates  to  actual  performance,  Xj.  The 
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system  developer  needs  to  develop  a  functional  form  that  accomplishes  this  translation. 
Both  linear  and  non-linear  functions  are  possible.  Additionally,  the  functions  may  be 
unique  for  each  i.  Equation  (7)  describes  the  translation  function: 

Equations  (6)  and  (7)  only  address  calculating  non-economic  performance 
attributes.  However,  an  underlying  reason  for  using  z;-  as  the  decision  variable  is  to 
facilitate  calculating  the  cost  of  an  alternative.  Therefore,  an  equation  is  needed  to 
translate  the  level  of  the  design  attribute,  zy,  into  a  cost  value. 

Cj  =  Cj  •  Zj  (8) 

Equation  (8)  calculates  Q,  the  cost  for  design  attribute  j,  by  multiplying  the  design 
attribute  level,  zy,  by  the  cost  factor  vector,  c  =(c\,...,  Cj,  c„),  which  is  indexed  to  j. 

The  cost  factors,  <y,  are  derived  from  collaboration  between  system  developers  and  cost 
estimators.  The  total  cost  for  an  alternative  is  calculated  from  the  following: 

m 

TC  =  ZCj  (9) 

j= 1 

As  demonstrated  by  Equation  (9),  the  total  cost  for  an  alternative,  TC,  is  the  sum  of  the 
costs  for  the  m  design  attributes,  C). 

Upon  deriving  a  means  to  determine  the  total  cost  for  an  alterative,  it  is  vital  for 
economic  factors  to  be  incorporated  into  the  overall  utility  function  described  by 
Equation  (2).  As  cited  in  previous  chapters,  guidance  on  CAIV  dictates  that  total  cost 
must  be  treated  as  a  measure  of  performance.  Thus,  the  overall  utility  of  an  alternative 
must  reflect  the  user’s  value  of  an  alternative’s  cost.  The  user  must  decide  on  the  shape 
the  SAU  function  specific  to  cost  PA.  Additionally,  a  scaling  constant  must  be  developed 
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for  the  cost  performance  attribute  that  indicates  the  user’s  willingness  to  make  cost  trade¬ 
offs.  As  a  measure  of  perfonnance,  TC  is  calculated  directly  from  zj.  Consequently,  it  is 
not  necessary  to  formulate  a  normalizing  translation  function,  like  the  one  specified  by 
Equation  (7)  for  this  performance  attribute.  Instead,  TC  is  directly  equivalent  to  x„  where 
i  is  the  index  specific  to  the  cost  PA. 

Having  defined  all  of  the  variables  necessary  to  fonnulate  the  core  CAIV  analysis 
model,  it  is  vital  to  discuss  the  subject  of  constraints.  The  core  model  employs  a  cost 
target  as  the  primary  system  constraint.  Based  upon  this  constraint,  the  model  seeks  to 
optimize  the  overall  utility  of  the  system  by  varying  the  levels  of  attainment  for  the 
individual  design  attributes.  Additional  side  constraints  for  technical  performance  may 
also  be  considered.  However,  because  the  SAU  functions  are  derived  from  the  threshold 
and  objective  levels  of  performance  specified  by  the  ORD,  there  is  the  potential  to  over 
constrain  the  system  by  adding  performance  side  constraints. 

Using  notation  presented  above,  it  is  possible  to  formulate  the  core  CAIV  analysis 

model  as  a  mathematical  program  seeking  to  optimize  the  overall  utility  of  the  weapon 

system  (Thurston  and  Locascio,  1994:64): 

maximize:  U(x)  (10) 

by  varying:  z 

subject  to:  x  =  g(z ) 

TC  <  TC 

Zj  min  —  Z  j  —  Z  j  max 

The  program  specified  by  Equation  (10)  uses  the  cost  target,  TCmax,  as  its  primary 
constraint.  TCmax  is  the  threshold  level  for  system  cost.  This  level  is  based  upon 
economic  resources  available  to  the  system  developer.  The  second  constraint  employed 
is  the  bounding  of  the  design  attribute  level,  zj,  between  zero  and  the  maximum  level  of 
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attainment,  z/max.  The  reasoning  for  this  constraint  has  been  described  above.  One  may 
then  solve  this  program  by  using  one  of  the  many  commercially  available  optimization 
applications. 

Initial  CAIV  analysis  begins  by  first  detennining  the  maximum  cost  required  to 
attain  ximax  for  all  perfonnance  attributes,  i,  except  for  total  cost.  The  total  alternative  cost 
resulting  from  this  solution  equates  to  the  cost  ceiling,  TCceuing ■  The  total  cost  ceiling  is 
interpreted  as  the  level  of  funding  beyond  which  no  additional  technical  perfonnance  is 
gained.  At  this  point,  all  non-economic  PA  are  maximized.  Next,  by  incrementally 
decreasing  the  cost  target,  TCmax,  from  the  cost  ceiling,  TCCeiiing,  and  then  solving  the  non¬ 
linear  program  specified  in  Equation  (10)  for  each  cost  increment,  it  is  possible  to 
understand  how  overall  utility  behaves  as  a  function  of  total  cost.  Figure  8  presents  a 
notional  depiction  of  this  behavior. 


Figure  8.  Notional  Depiction  of  Overall  Utility  as  a  Function  of  Total  Cost 

Evaluating  overall  utility  as  a  function  of  cost  allows  one  to  assess  the  marginal 
benefit  (or  detriment)  incurred  by  incrementally  increasing  (or  decreasing)  the  cost  target 
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from  its  current  level.  Such  analysis  helps  to  identify  the  “knee  of  the  curve;”  the  point  at 
which  each  marginal  increase  in  utility  becomes  increasingly  expensive  (Lorell  and 
Graser,  2001:34).  It  is  also  possible  to  examine  the  individual  solutions  for  eachx,  and  Zj. 
Such  analysis  reveals  how  the  individual  perfonnance  attributes  behave  and  how  the 
economic  resources  are  distributed  to  each  of  the  design  attributes  as  the  total  cost 
constraint  changes. 

Expanded  CAIV/EA  Model 

Having  established  the  foundations  for  the  core  CAIV  model,  it  is  now  possible  to 
begin  integrating  features  specific  to  EA.  As  it  stands,  the  core  CAIV  model  seeks  to 
optimize  the  overall  utility  for  a  single  system  development  activity.  The  core  model  is 
consistent  with  a  “single  step”  acquisition  strategy.  Using  a  single  step  approach,  the 
user  gains  benefits  from  the  system  (in  terms  of  its  technical  performance  and 
capabilities)  only  after  the  development  activity  is  complete.  The  limitations  associated 
with  this  approach  have  been  explained  previously.  An  EA  approach  seeks  to  overcome 
these  limitations  by  incrementally  delivering  to  the  user  solutions  to  operational 
requirements  (i.e.,  capabilities,  MOPs,  etc.),  over  time.  Along  the  way,  an  EA  approach 
mitigates  technical  risk  and  uncertainty  by  delivering  lower  risk  requirements  in  earlier 
increments  and  higher  risk  requirements  in  later  ones  (after  the  risk  has  been  reduced  by  a 
combination  of  spiral  development  activities).  Thus,  the  expanded  CAIV/EA  model 
needs  to  incorporate  two  additional  features:  time-phasing  and  technical  risk  mitigation. 

Incremental  delivery  of  capability  and  requirements  phasing  introduce  an 
additional  dimension  into  the  model,  time.  EA  increments  are  assumed  to  occur 
sequentially  over  time.  While  there  may  be  some  overlap  between  two  increments, 
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general  practice  dictates  that  completion  of  one  increment  does  not  occur  until  the 
completion  of  the  preceding  increment  (except  for  the  first,  or  core  increment,  which  has 
no  predecessor).  Thus,  the  expanded  CAIV/EA  model  treats  each  increment  as  a  discrete 
development  activity. 

The  expanded  model  dimensionality  is  implemented  through  the  addition  of  the 
subscript  variable  /,  which  augments  some  of  the  core  model  parameters.  In  many  cases, 
this  addition  changes  many  variables,  which  were  previously  described  as  vectors,  into 
matrices.  For  example,  the  perfonnance  attribute  variable,  xu,  represents  the  level  of  the 
/th  performance  attribute  for  the  /th  increment.  The  total  design  attribute  variable,  Zj,, 
represents  the  total  level  of  the  /th  design  attribute  for  the  /th  increment.  In  all  cases,  the 
variable,  /,  equals  the  1,2,  ...  , p  increments. 

With  regards  to  the  total  design  attribute  variable,  Zju  there  is  a  reason  for 
capitalizing  the  letter  “z.”  Capitalization  differentiates  the  total  level  of  the  design 
attribute  from  the  incremental  level  of  the  design  attribute,  Zy/.  The  incremental  design 
attribute  variable,  z7y,  represents  the  level  of  the  /th  design  attribute  for  the  /th  increment. 
Because  an  EA  strategy  centers  on  increasing  a  systems  capability  over  a  series  of 
increments,  it  is  assumed  that  the  latter  increments  build  upon  the  work  accomplished 
during  earlier  ones.  Thus,  z7/  represents  the  marginal  increase  in  DA  accomplished  in  a 
given  increment.  Whereas,  2/7  represents  cumulative  level  of  a  design  attribute  for  a 
given  increment,  taking  into  account  the  present  increment’s  marginal  increase  as  well  as 
prior  increments’  levels  of  attainment.  The  Equation  (11)  describes  the  relationship 
between  the  total  and  incremental  levels  of  DA  attainment. 

Zj,  Z  ,  i !  Zjj  (n) 
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The  previous  discussion  has  begun  to  integrate  EA  time -phasing  characteristics 
with  the  core  CAIV  model.  However,  before  this  feature  can  be  fully  addressed,  it  is 
essential  to  discuss  the  other  aspect  of  EA,  technical  risk  mitigation.  As  was  described  in 
previous  chapters,  EA  embraces  the  concept  that  capabilities  with  lower  technical  risk 
should  be  delivered  earlier  in  the  development  cycle  than  higher  risk  capabilities. 
Additionally,  an  EA  approach  (employing  spiral  development  techniques)  suggests  that 
technical  risk  can  be  reduced  over  the  course  of  the  development  cycle  through  an 
iterative  process  of  systems  engineering,  experimentation,  operational  evaluation,  and 
user  feedback  (see  Figure  2).  Thus,  the  degree  of  technical  risk  should  reduce  as  the 
development  cycle  proceeds  from  earlier  to  later  increments. 

Risk  is  incorporated  into  the  model  with  the  matrix  Djj,  the  degree  of  technical 
risk  associated  with y'th  design  attribute  for  the  /th  increment.  The  technical  risk 
parameter  discounts  the  level  of  the  incremental  design  attribute  variable,  zji,  and  thus 
affects  the  level  of  the  total  design  attribute  variable,  Z/7,  as  well.  Values  for  Djt  range 
from  zero  (easily  attained)  to  one  (impossible  to  attain).  The  relationship  between  the 
technical  risk  variable  and  the  design  attribute  variables  is  expressed  by  modifying 
Equation  (11). 

Z  j,i  =  Zj,i- 1  +  (l  -  Dj,i)-  Zj,i  (12) 

The  technical  risk  factor  discounts  the  realized  attainment  for  an  incremental  design 
attribute  variable.  Thus,  the  incremental  design  attribute  variable  can  be  considered  a 
planned  level  of  attainment,  while  the  product  of  the  incremental  DA  variable  and  the 
technical  risk  factor  is  equivalent  to  the  actual  level  of  attainment.  This  formulation 
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implies  that  higher  risk  activities  realize  inferior  levels  of  DA  attainment  than  lower  risk 
activities. 

As  was  cited  previously,  the  EA  approach  assumes  that  technical  risk  decreases  as 
the  development  cycle  proceed  from  earlier  to  later  increments.  Therefore,  the  expanded 
CAIV/EA  model  must  include  the  following  assumption: 

D,r  D._  03) 

Equation  (13)  indicates  that  the  technical  risk  factor  for  an  increment  must  be  less  than  or 
equal  to  the  technical  risk  factor  for  the  preceding  development  increment,  for  a  given 
design  attribute.  The  values  for  the  technical  risk  factors  are  derived  from  expert  opinion 
and  input  from  the  various  system  development  IPTs.  The  degree  by  which  the  technical 
risk  factors  reduce  over  time  should  be  based  upon  the  level  and  extent  of  risk  mitigation 
activities  being  accomplished  as  part  of  the  spiral  development  process.  Again,  this 
assessment  needs  to  be  made  by  those  who  are  involved  with  the  system  development 
IPTs. 

Having  integrated  technical  risk  mitigation  into  the  expanded  CAIV/EA  model,  it 
is  now  possible  to  finish  integration  of  the  time-phasing  component.  There  are  additional 
core  parameters  that  are  affected  by  the  expansion  of  the  time  dimension.  The  elements 
of  the  cost  factor  matrix,  Cjj,  represent  the  cost  factors  associated  with  /th  design  attribute 
for  the  /th  increment.  As  a  result  of  this  change,  the  elements  of  the  design  attribute  cost 
matrix,  Q/,  represent  the  cost  of  the  /th  design  attribute  for  the  /th  increment.  The 
rationale  for  variation  in  the  cost  factor  term  is  based  upon  learning  curve  improvements 
and  other  efficiencies,  which  often  result  as  the  development  cycle  proceeds  forward  into 
time.  The  incremental  total  cost  is  expressed  by  the  vector  tc  =  (tc\, tci,  tcp),  where 
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tci  represents  an  alternative’s  total  cost  at  the  /th  increment.  Finally  the  overall  total  cost 
is  calculated  from  the  following: 

TC  =  ^tCi  (14) 

/= l 

The  overall  total  cost  for  an  alternative,  as  calculated  by  Equation  (14),  sums  the 
incremental  total  costs  across  the p  increments. 

For  simplicity’s  sake,  the  expanded  CAIV/EA  model  assumes  that  the  scaling 
constants,  kj,  remain  the  same  from  one  increment  to  the  next  (at  least  for  an  initial 
iteration  of  the  model).  Along  this  line,  it  is  also  assumed  that  the  single  attribute  utility 
functions  do  not  vary  from  one  increment  to  another.  However,  because  of  the  recursive 
nature  of  EA,  it  is  quite  possible  that  the  data  for  these  two  components  will  require 
updates  as  the  development  cycle  proceeds.  Upon  completion  of  an  actual  development 
increment,  it  is  likely  that  the  user  will  have  new  guidance  regarding  their  willingness  to 
make  trade-offs  between  perfonnance  attributes  (hence  the  need  to  modify  the  scaling 
constants).  Regardless  of  the  situation,  it  is  vital  to  keep  the  utility  data  current  and  in¬ 
line  with  user  preferences.  Finally,  the  elements  of  the  performance/design  attribute 
relationship  matrix,  Ry,  are  assumed  to  remain  constant  from  one  increment  to  another. 

There  remain  two  more  additions  to  the  core  CAIV  model.  First,  the  notation  for 
the  overall  utility  function  must  be  modified  to  reflect  the  utility  associated  with  a  given 
increment. 


]_ 

K 


i= 1 


1 


(15) 
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Equation  (15)  calculates  the  incremental  utility  vector,  u  =  (u\, ui  up),  for  the  /th 
increment.  There  exists  an  incremental  utility  function  for  each  of  the  p  increments.  The 
second  and  final  modification  is  the  creation  of  a  new  overall  utility  function.  It  is 
necessary  to  express  the  overall  utility  as  a  function  of  the  individual  utilities  for  each  of 
the  increments.  Additionally,  it  is  necessary  to  incorporate  the  user’s  preference  for  the 
character  of  the  development  cycle  schedule.  These  schedule  preferences  are  specified 
by  the  schedule  weighting  factors. 

P 

U(u)  =  ^srui  (16) 

1=1 

The  overall  utility,  U,  as  calculated  from  Equation  (16)  is  the  sum  of  the  incremental 
utilities,  w/,  each  multiplied  by  their  respective  schedule  weighting  factor,  s/.  The  values 
for  the  schedule  weighting  factors  sum  to  unity.  Therefore,  the  larger  an  increment’s 
schedule  weighting  factor,  the  more  emphasis  is  placed  on  increasing  the  utility  for  that 
increment.  Because  the  increments  are  assumed  to  occur  sequentially,  the  schedule 
weighting  factors  dictate  the  nature  of  the  development  cycle  (i.e.,  the  factors  suggest  the 
increment,  and  thus  the  point  in  time,  where  the  preponderance  of  weapon  system 
capability  is  delivered). 

Based  upon  the  changes  described  above,  it  is  now  possible  to  formulate  the 

expanded  CAIV/EA  model  as  a  non-linear  program. 

maximize:  U(u)  (17) 

by  varying:  z 

subject  to:  u  =  u(x/) 

xi  =  g(Zl) 

TC  <TCmax 
7  <7  <7 

t-'jmm  —  —  ^-jinax 
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While  the  program  described  by  Equation  (17)  closely  resembles  the  one  used  in  the  core 
CAIV  model,  there  are  two  important  differences.  First,  the  objective  function  has  been 
modified  to  optimize  the  overall  utility  for  each  of  the  p  development  increments. 

Second,  the  decision  variable,  zp,  represents  the  planned  marginal  increase  in  the  level  of 
attainment  for  the  design  attribute.  This  is  a  significant  difference  from  Equation  (10), 
where  the  decision  variable  was  the  realized  level  of  attainment  for  a  design  attribute. 
These  two  differences  relate  directly  back  to  the  two  underlying  themes  of  EA,  time¬ 
phasing  and  technical  risk  mitigation.  The  remainder  of  the  expanded  CAIV/EA  model  is 
consistent  with  the  one  described  in  the  previous  section.  The  total  cost  for  an 
alternative,  TC,  is  bounded  by  the  cost  target,  TCmax.  Finally,  the  cumulative  level  of 
(realized)  attainment  for  a  design  attribute,  Zji,  is  bounded  by  zero  and  the  maximum 
level  of  attainment  for  that  attribute,  Zjmax. 

The  analytical  approach  described  for  the  core  model  remains  valid  for  the 
expanded  CAIV/EA  model.  By  first  identifying  the  cost  ceiling,  and  then  incrementally 
reducing  the  cost  target,  TCmax,  from  the  ceiling,  it  is  possible  to  understand  how  the 
overall  utility  (as  well  as  incremental  utility)  behaves  as  a  function  of  cost. 

Model  Evaluation  Techniques 

Having  formulated  the  CAIV/EA  model,  it  is  important  to  determine  whether  or 
not  it  accurately  reflects  the  integration  of  CAIV  analysis  within  an  EA  framework. 
According  to  Law  and  Kelton,  one  of  the  most  difficult  problems  in  modeling  is  trying  to 
determine  whether  a  model  is  an  accurate  representation  of  the  actual  system  being 
studied,  i.e.,  whether  the  model  is  valid  (Law  and  Kelton,  2000:264).  Validation  is  the 
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process  of  determining  whether  a  model  is  an  accurate  representation  of  the  system,  for 
the  particular  objectives  of  the  study  (Law  and  Kelton,  2000:265). 

Law  and  Kelton  assert  that  the  most  definitive  test  of  a  model’s  validity  is  to 
establish  that  its  output  data  closely  resemble  the  output  data  that  would  be  expected  from 
the  actual  system  (Law  and  Kelton,  2000:279).  Unfortunately,  there  are  several  facets 
that  make  validation  of  the  CAIV/EA  model,  in  this  way,  a  challenging  proposition.  The 
CAIV/EA  model  is  intended  to  serve  primarily  as  a  planning  tool  for  weapon  system 
development.  Thus,  by  definition,  the  system  which  is  being  modeled  is  nonexistent. 
Consequently,  it  is  not  possible  to  compare  the  outputs  from  the  CAIV/EA  model  to 
those  from  an  actual  development  program.  Additionally,  the  recent  guidance  from 
USD(AT&L)  on  the  subject  of  CAIV/EA  planning  means  there  are  presently  no 
documented  models  available  which  might  serve  as  benchmarks  for  comparison.  Faced 
with  these  obstacles,  how  might  one  attempt  to  validate  the  CAIV/EA  model? 

Law  and  Kelton  offer  some  suggestions  when  approaching  validation  of  a  model 
for  a  nonexistent  system.  They  suggest  a  fonn  of  “concurrent  validation”  where 
validation  takes  place  in  concert  with  the  development  of  the  model.  This  concurrent 
validation  relies  upon  a  combination  of  management  involvement,  subject  matter  expert 
(SME)  opinion,  and  sensitivity  analysis  (Law  and  Kelton,  2000:274-8).  Fortunately,  all 
of  these  components  are  available  to  assist  in  validating  the  CAIV/EA  model. 

Validation  of  the  CAIV/EA  model  will  occur  via  case  study  and  will  employ  a 
“concurrent  validation”  approach.  The  notional  ground  based  C2  system,  will  serve  as  a 
test  case  for  this  study.  The  test  case  system  is  analogous  to  a  real-world  Air  Force 
development  program,  currently  at  an  early  point  in  its  development  cycle.  The  program 
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office  managing  the  analog  system  is  concerned  about  delivering  the  user  an  optimal 
level  of  perfonnance  while  balancing  budgetary  constraints;  thus  making  it  a  good 
candidate  for  CAIV  analysis.  Additionally,  the  analogous  system  is  a  complicated, 
software-intensive  command  and  control  system.  As  explained  by  the  previous  chapter, 
these  characteristics  suggest  employing  an  EA  strategy.  In  light  of  the  USD(AT&L) 
guidance  on  CAIV/EA  planning,  the  very  nature  of  the  test  case  system  makes  it  a 
suitable  test  case  for  evaluation  of  the  CAIV/EA  model. 

CAIV/EA  model  validation  will  begin  with  close  interaction  with  the  Air  Force 
program  management  office.  This  interaction  will  help  to  provide  a  better  understanding 
of  the  system,  its  architecture,  and  other  characteristics  important  to  the  CAIV/EA  model. 
Then,  working  with  technical  experts  and  cost  estimators,  the  specific  data  required  by 
the  CAIV/EA  model  will  be  collected.  Care  will  be  taken  to  ensure  that  the  data 
accurately  reflects  the  nature  of  the  test  case  system,  as  it  is  understood  by  those  who 
have  the  greatest  knowledge  of  it.  Next,  the  model  will  be  exercised  and  the  output  data 
will  be  collected.  Appropriate  analysis  will  be  perfonned  to  glean  specific  infonnation 
from  the  data  (such  as  the  behavior  of  overall  and  incremental  utility  as  a  function  cost, 
performance  attribute  levels  as  a  function  of  cost,  etc.).  Additional  sensitivity  analysis 
will  be  performed  to  help  determine  which  model  inputs  and  parameter  have  a  significant 
impact  upon  the  model  outputs.  This  sensitivity  analysis  will  help  to  determine  which 
model  inputs  need  to  be  modeled  more  carefully  (Law  and  Kelton,  2000:278). 

Luman  also  suggest  of  a  series  of  challenges  to  be  wary  of  when  implementing  a 
CAIV  model.  While  his  suggestions  were  specific  to  the  “System  of  Systems”  CAIV 
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model,  they  are  also  pertinent  to  the  CAIV/EA  model.  To  reiterate,  Luman  cited  the 
following  as  being  important  to  the  CAIV  model  validation  process: 


•  Defining  the  overarching  MOE, 

•  Allocation  of  system  components  and  selection  of  trade  space  for 
MOPs, 

•  Adaptation/adoption  of  appropriate  perfonnance  based  cost  models, 
and 

•  Application  of  efficient  and  appropriate  optimization  algorithms 
(Luman,  1999:11). 

In  the  context  of  the  test  case,  the  CAIV/EA  model  validation  process  will  address  each 
of  these  points  to  increase  the  validity  of  the  model. 
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IV.  Results  and  Analysis 


Overview 

This  chapter  begins  with  a  description  of  the  notional  C2  system  test  case.  Next, 
the  particulars  of  the  test  case  are  integrated  with  the  CAIV/EA  model  developed  in  the 
previous  chapter.  Several  ground  rules  and  assumptions  are  established  to  frame  and 
assist  the  ensuing  analysis.  Next,  the  model  is  exercised,  output  data  is  collected,  and 
preliminary  analysis  is  performed.  Based  upon  this  initial  analysis,  several  questions 
pertaining  to  the  behavior  of  the  model  are  raised.  Finally,  additional  sensitivity  analysis 
is  accomplished  to  better  describe  how  the  CAIV/EA  model  responds  to  variations  in  its 
input  parameters. 

Test  Case  Description  and  Model  Integration 

As  was  mentioned  in  the  previous  chapter,  a  notional  ground  based  C2  system, 
serves  as  a  test  case  for  this  study.  The  notional  system  is  intended  to  support  air  and 
space  battle  management  and  execution  functions  including  data  link  management, 
surveillance,  identification,  and  air  battle  execution  for  North  American  aerospace 
defense.  The  system  is  to  provide  surveillance  and  control  of  US  airspace  (including 
counter  drug  detection  and  monitoring  operations),  warning  and  assessment  of  aerospace 
attack,  and  response  against  air  attack.  The  system  shall  also  monitor  airborne  activity  in 
support  of  North  American  Aerospace  Defense’s  (NORAD)  homeland  defense  (HLD), 
air  sovereignty,  and  aerospace  defense  missions  within  its  Area  of  Responsibility  (AOR) 
on  a  continuous,  uninterrupted  basis.  Additionally,  the  system  is  to  provide  effective  and 
integrated  battle  management  of  aerospace  defense  resources  during  peacetime, 
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transition,  attack,  and  post-attack  periods.  The  system  shall  process,  integrate,  display, 
and  distribute  data  from  sensors,  data  links,  and  other  C2  agencies  to  maintain  situational 
awareness  and  support  air  interdiction  operations. 

Based  upon  this  description  of  the  notional  C2  system,  the  following  technical 
(i.e.,  non-economic)  measures  of  performance  are  derived  and  listed  below: 


Table  3.  C2  System  Technical  Measures  of  Performance 


System  Administration 

Human-Machine  Interface 

Aerospace  Surveillance 

Target  Identification 

Weapons  and  Battle  Management* 

Tactical  Data  Links* 

Training  and  Simulation 

System  Load  Capacity 

The  addition  of  the  economic  measure  of  performance  to  this  list  (i.e.,  system  cost) 
increases  the  total  number  of  MOPs  to  nine  (9).  Thus,  in  terms  of  the  notation  presented 
in  the  previous  chapter,  there  are  nine  (9)  elements  of  the  performance  attribute  vector* 
(, n  =  9).  The  ensuing  table  presents  the  relevant  data  associated  with  the  performance 
attribute  vector: 


Table  4.  C2  System  Performance  Attribute  Vector  Details 


/ 

Name 

Short  Name 

Units 

Value  Ranoe 

1 

System  Administration 

Sys  Admin 

%  Implemented 

0:1 

2 

Human-Machine  Interface 

HMI 

%  Implemented 

0:1 

3 

Aerospace  Surveillance 

Surveillance 

%  Implemented 

0:1 

4 

Target  Identification 

Identification 

%  Implemented 

0:1 

5 

Weapons  and  Battle  Management 

W&BM 

%  Implemented 

0:1 

6 

Tactical  Data  Links 

Data  Links 

%  Implemented 

0:1 

7 

Training  and  Simulation 

Trng/Sim 

%  Implemented 

0:1 

8 

System  Load  Capacity 

Sys  Load 

%  Implemented 

0:1 

9 

System  Cost 

Cum  Cost 

$ 

0:Cost  Ceiling 

Table  4  explains  each  element  (/)  of  the  perfonnance  attribute  vector*.  Each 
element  is  described  by  its  full  name,  a  shortened  name  (for  identification  purposes 
during  analysis),  its  unit  of  measurement,  and  finally  by  its  range  of  valid  values.  For 
example,  System  Administration  (or  Sys  Admin)  is  associated  with  element  one  (*i)  of 
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the  performance  attribute  vector.  The  MOP  is  measured  in  tenns  of  its  relative  degree  of 
implementation.  Based  upon  this  designation,  valid  values  for  jci  lie  between  zero  (0)  and 
(1).  From  this  definition,  it  is  inferred  that  zero  percent  implementation  means  that  the 
MOP  (and  its  associated  capability)  has  not  been  implemented  or  addressed  at  all. 
Conversely,  100  percent  implementation  means  that  the  MOP  has  been  fully  and 
completely  implemented. 

The  CAIV/EA  model  does  not  require  all  values  for  the  performance  attribute 
matrix  to  lie  within  this  range.  In  fact,  the  valid  range  for  X9  (System  Cost)  is  between 
zero  (0)  and  presumably  some  number  much  greater  than  one  (1).  Because  elements  x\ ... 
x 8  are  primarily  descriptive  in  nature  (as  opposed  to  being  measured  and  quantified)  the 
decision  to  use  a  relative  scale  was  based  on  the  difficulty  associated  with  establishing  a 
relevant  metric  for  each.  Additionally,  it  is  convenient  to  translate  the  nonnalized  level 
of  performance  (*f  )  calculated  from  Equation  6  into  the  actual  level  of  performance  by 
selecting  a  translation  function  ( hi  [xf] )  that  returns  a  value  of  x\  between  0  and  1 .  A 
more  complete  explanation  of  the  translation  functions  used  in  this  analysis  follows 
shortly. 

For  the  purposes  of  this  analysis,  it  is  assumed  that  the  EA  strategy  specified  for 
the  notional  C2  system  is  to  be  accomplished  over  the  course  of  three  increments  (p  =  3). 
In  accordance  with  the  guidance  on  EA  described  in  Chapter  II,  each  of  these  increments 
is  intended  to  represent  approximately  18  months  in  time.  Additionally,  the  increments 
are  arranged  serially,  in  ascending  order. 
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Table  5  specifies  some  additional  data  relevant  to  each  of  the  MOPs.  The  table 


presents  the  performance  function  parameter  (PFP),  utility  function  parameter  (UFP),  and 
scaling  constant  for  each  i. 


Table  5.  Baseline  MOP  Model  Parameter  Specification 


MOP  Factors 

PFP  UFP  ki 

MOP  Name: 

1 

Sys  Admin 

1.00 

1.00 

0.10 

2 

HMI 

1.00 

1.00 

0.10 

3 

Surveillance 

1.00 

1.00 

0.10 

4 

Identification 

1.00 

1.00 

0.10 

5 

W  &  BM 

1.00 

1.00 

0.10 

6 

Data  Links 

1.00 

1.00 

0.10 

7 

Trng/Sim 

1.00 

1.00 

0.10 

8 

Sys  Load 

1.00 

1.00 

0.10 

9 

Cum  Cost 

NA 

1.00 

0.10 

The  performance  function  parameter  describes  the  shape  of  the  translation  function  which 
converts  the  normalized  level  of  performance  (xf  )  into  the  corresponding  performance 
attribute  vector  element.  The  performance  translation  function  used  in  this  analysis 
derived  from  an  approximation  to  the  cumulative  distribution  function  (CDF)  for  the 
standard  Beta  distribution  when  a  =  1  (using  the  MS  Excel  Betadist  function).  The  PFP 
controls  the  [1  parameter  used  in  the  Beta  distribution. 

Figure  9  illustrates  the  effect  of  PFP  selection  upon  the  translation  from  relative  to 
absolute  performance.  This  function  behaves  similarly  to  the  utility  functions  described 
in  Figure  6  of  Chapter  II.  In  fact,  the  utility  function  parameter  (UFP)  is  used  in  the  same 
manner  as  the  PFP.  The  UFP  also  equates  to  the  [5  parameter  used  in  the  standard  Beta(a 
=  1)  distribution  (however,  each  MOP  can  have  different  values  for  their  respective  PFP 
and  UFP).  Finally,  kt  is  the  single  perfonnance  attribute  scaling  constant  for  MOP  i.  As 
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was  described  in  the  previous  chapter,  ki  represents  the  decision  maker’s  willingness  to 
make  trade-offs  with  the  specified  MOP. 

Various  Performance  Translation  Functions  j 

0)  1.20 
o 

<5  1.00 

E 

'o  0.80 
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Figure  9.  The  Effect  of  PFP  Selection  upon  Performance  Translation 
At  the  start  of  this  analysis,  all  UFP  and  PFP  values  have  been  set  to  one  (1). 
Additionally,  all  of  the  scaling  constants  have  been  set  to  0.10.  The  resulting  overall 
utility  scaling  constant  ( K)  equals  0.261  (via  Equation  2).  This  configuration  of 
parameters  indicates  that  there  is  a  positive  linear  relationship  between  relative  and 
absolute  performance.  There  is  also  a  positive  linear  relationship  between  absolute 
performance  and  single  attribute  utility  (for  technical  MOPs).  Because  a  decision  maker 
would  most  likely  have  less  value  for  more  expensive  alternatives,  the  single  attribute 
utility  function  for  the  economic  MOP  is  adjusted  (by  one  (1)  minus  the  resulting  utility) 
to  create  a  negative  linear  relationship  between  system  cost  and  single  attribute  utility. 
Additionally,  with  all  of  the  single  perfonnance  attribute  scaling  constants  set  equal  to 
each  other,  there  is  equal  willingness  to  trade-off  the  MOPs  when  calculating  overall 
utility  via  Equation  2.  The  elements  of  the  schedule  weighting  vector,  s,  are  also  given 
equivalent  weights  (approximately  0.333)  to  add  further  parity. 
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As  it  stands,  the  model  configuration  expressed  in  Table  5  would  most  likely  not 
be  consistent  with  a  decision  maker’s  true  value  system  and  trade-off  preferences. 
However,  this  implementation  provides  a  baseline  which  can  be  exercised,  evaluated,  and 
then  calibrated  to  better  reflect  the  decision  maker’s  preferences. 

Table  6.  Test  Case  System  Design  Attributes 


j  Reqt  #  Zimax  Units 

1 

4.1. 1.1 

2 

4.1. 1.2 

3 

4 

4.1. 1.4 

5 

SEZKA 

6 

ssms 

7 

am 

8 

9 

ams 

anas— 

11 

ES 

12 

grants 

13 

mamm 

14 

4.1.3.10 

15 

4.1.3.11 

16 

4.1.3.12 

The  previous  section  looked  only  at  the  performance  attributes  of  the  test  case 
system.  Now  it  is  time  to  consider  the  notional  C2  system’s  design  attributes;  those 
elements  that  the  decision  maker  controls  and  affects  (i.e.,  the  decision  variables). 
Because  the  notional  C2  system  is  software  intensive,  the  system’s  functional 
requirements  are  treated  as  the  decision  variables  for  the  CAIV/EA  model.  Table  6 
presents  the  sixteen  (16)  design  attributes  (m  =  16)  considered  in  this  evaluation.  Table  6 
also  presents  each  design  attribute’s  reference  number  (the  citation  that  would  identify 
the  functional  requirement  in  the  system’s  technical  requirements  document  (TRD)).  The 
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column  entitled  “Z/max”  describes  the  maximum  level  of  attainment  required  to  fully 
implement  the  corresponding  design  attribute.  The  final  column  describes  the  unit  of 
measure  for  each  design  attribute.  As  has  already  been  mentioned,  the  decision  variables 
for  this  analysis  are  the  various  functional  requirements  implemented  through  software 
coding.  Thus  the  appropriate  units  for  all  of  the  attributes  are  source  lines  of  code 
(SLOC). 


Table  7.  C2  System  QFD  Matrix 


Perl 

Formance  Attributes  (MOP)  /  i 

1 

2 

3 

4 

5 

6 

7 

8 

Design  Attributes 

29 

44 

41 

32 

36 

42 

25 

56 

:Col  Sum 

j  TRD  Req#  SLOC 

c 

E 

■u 

< 

IA 

>■ 

(/) 

Ml 

Z 

X 

u 

u 

c 

_ro 

"33 

£ 

3 

(/) 

c 

o 

X 

to 

u 

JC 

X 

c 

<u 

-o 

1— 1 

Z 

GQ 

08 

5 

U1 

-S£ 

C 

□ 

(0 

4-> 

to 

Q 

E 

55 

Ol 

c 

L. 

H 

■u 

to 

o 

_l 

(A 

<?T 

Row  Sum: 

1  4.1. 1.1 

2800 

1 

0 

3 

3 

1 

3 

i 

l 

13 

2  4.1. 1.2 

2800 

0 

3 

3 

3 

9 

3 

0 

3 

24 

3  4.1. 1.3 

700 

1 

1 

1 

3 

1 

3 

0 

3 

13 

4  4. 1.1.4 

700 

0 

0 

3 

1 

3 

1 

1 

9 

18 

5  4. 1.3.1 

2700 

1 

3 

1 

1 

3 

9 

1 

3 

22 

6  4. 1.3. 2 

2700 

1 

1 

9 

1 

1 

1 

0 

3 

17 

7  4. 1.3. 3 

2700 

3 

3 

3 

1 

1 

3 

1 

1 

16 

8  4. 1.3. 4 

2700 

3 

1 

3 

1 

1 

3 

0 

9 

21 

9  4. 1.3. 5 

720 

1 

9 

1 

1 

3 

3 

1 

1 

20 

10  4. 1.3. 6 

540 

3 

1 

1 

3 

3 

3 

0 

3 

17 

11  4. 1.3. 7 

540 

0 

3 

3 

1 

0 

1 

1 

3 

12 

12  4. 1.3. 8 

540 

1 

1 

0 

1 

3 

1 

3 

3 

13 

13  4. 1.3. 9 

540 

3 

3 

3 

1 

3 

3 

1 

9 

26 

14  4.1.3.10 

540 

1 

3 

3 

9 

3 

3 

9 

1 

32 

15  4.1.3.11 

540 

9 

9 

3 

1 

0 

1 

3 

3 

29 

16  4.1.3.12 

540 

1 

3 

1 

1 

1 

1 

3 

1 

12 

Having  described  both  the  performance  and  design  attributes,  it  is  now  possible  to 
relate  the  two  via  a  QFD  matrix.  Table  7  presents  the  matrix  created  by  placing  the 
performance  and  design  attribute  vectors  orthogonal  to  one  another.  The  resulting  matrix 
is  dimensioned  by  the  number  of  technical  MOPs  and  the  number  of  design  attributes 
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(8x16)  (for  formatting  purposes,  the  matrix  shown  above  has  been  turned  90-degrees  to 
place  the  design  attributes  on  the  vertical  axis).  The  elements  of  matrix  R  have  been 
populated  using  a  relationship  scale  ranging  from  zero  (0)  to  nine  (9).  These  elements 
represent  the  strength  of  the  relationship  between  each  pair  of  performance  and  design 
attributes.  A  value  of  zero  (0)  indicates  no  relationship  exists  between  the 
performance/design  attribute  pair.  A  value  of  one  (1)  indicates  “some”  positive 
relationship  exists.  A  value  of  three  (3)  represents  a  “strong”  relationship,  three  times 
stronger  than  a  value  of  one.  A  nine  (9)  is  indicative  of  a  “very  strong”  relationship, 
three  times  stronger  than  a  value  of  three.  While  the  values  for  the  performance  attribute 
model  parameters  will  be  adjusted  over  the  course  of  the  analysis,  the  values  found  in 
Table  7  will  remain  constant. 


Table  8.  C2  System  Risk  Matrix 


j  Reqt  #  Inc  1  Risk  Inc  2  Risk  Inc  3  Risk 

m 

E»W« 

0.13 

0.01 

0.01  I 

2 

EH  m 

0.31 

0.06 

0.03 

3 

EH1H 

0.13 

0.05 

0.00 

4 

4.1. 1.4 

0.33 

0.28 

0.24 

5 

4.1. 3.1 

0.38 

0.32 

0.03 

6 

e»pc wjm 

0.26 

0.05 

0.02 

7 

Ezsm 

0.08 

0.05 

0.02 

8 

4.1. 3.4 

0.24 

0.00 

0.00 

9 

4.1. 3. 5 

0.02 

0.00 

0.00 

10 

4.1. 3.6 

0.12 

0.10 

0.03 

11 

ESSSB 

0.05 

0.05 

0.01 

12 

4.1. 3.8 

0.24 

0.05 

0.02 

13 

USES! 

0.33 

0.31 

0.06 

14 

4.1.3.10 

0.24 

0.05 

0.01 

15 

4.1.3.11 

0.37 

0.05 

0.02 

16 

4.1.3.12 

0.13 

0.00 

0.00 

Establishing  the  QFD  matrix  enables  one  to  translate  a  system  alternative’s 
performance  from  its  design.  However,  it  is  also  necessary  to  be  able  to  calculate  the 
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realized  level  of  attainment  from  the  planned  level  of  attainment  for  a  given  design 
attribute.  Thus,  to  implement  Equation  12  it  is  necessary  to  create  the  risk  matrix  D. 
Table  8  presents  the  16  x  3  matrix  generated  by  meshing  the  design  attributes  with  the 
three  development  increments.  The  values  for  each  of  the  elements  in  the  risk  matrix 
have  the  potential  to  range  zero  (0)  to  one  (1).  However,  in  this  test  case  all  of  the  risk 
has  been  assessed  to  be  below  0.4.  Examination  of  the  matrix  also  reveals  that  as  the 
development  progresses  from  earlier  toward  later  increments,  the  risk  associated  with  any 
given  design  attribute  decreases.  This  is  consistent  with  the  assumption  made  in  the 
previous  chapter  that  risk  will  decrease  over  time  (due  to  technology  maturation  and  risk 
mitigation  efforts).  Just  as  with  the  QFD  matrix  in  Table  7,  the  values  of  the  elements  of 
the  risk  matrix  in  Table  8  will  remain  constant  over  the  course  of  the  ensuing  analysis. 


Analysis  Ground  Rules  and  Assumptions 

Beyond  the  data  cited  in  the  previous  section,  some  additional  clarification  is 
required  to  facilitate  the  upcoming  analysis.  The  following  is  a  list  of  the  major 
analytical  ground  rules  and  assumptions: 

•  Although  SLOC  are  technically  discrete  quantities,  the  values  for  the  elements 
of  the  incremental  design  attribute  vectors  are  considered  continuous  across 
their  respective  feasible  ranges.  The  values  for  each  Z-jmax  are  sufficiently 
large.  Thus,  there  is  little  value  in  mandating  integer  values  for  each  element 
of  the  incremental  design  attribute  vectors. 

•  The  software  cost  factor  is  $89.52  per  SLOC. 

•  The  only  costs  considered  by  the  model  are  those  associated  with  the  design 
attributes.  While  there  would  undoubtedly  be  additional  costs  associated  with 
the  development  program  (e.g.,  Systems  Engineering  /  Program  Management, 
Test,  Data,  etc.),  this  analysis  only  considers  the  direct  costs  associated  with 
the  design  alternative. 
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•  For  all  MOPs  and  their  corresponding  single  performance  attribute  utility 
functions,  the  absence  of  perfonnance  attainment  translates  to  a  utility  value 
of  zero  (0).  Likewise,  the  objective  level  of  the  performance  attribute 
translates  to  a  utility  value  of  one  (1). 

•  To  simplify  the  analysis,  the  threshold  and  objective  levels  for  each  measure 
of  performance  in  each  increment  are  held  constant  and  equivalent.  For 
technical  MOPs,  the  threshold  level  occurs  at  0%  implementation  and  the 
objective  level  occurs  at  100%  implementation.  For  the  economic  MOP,  the 
objective  level  corresponds  to  a  system  cost  of  $0.00  while  the  threshold  level 
exists  as  the  maximum  design  cost. 


Initial  Analysis 

Using  the  data  presented  in  the  previous  section,  the  notional  C2  system  test  case 
has  been  implemented  in  a  spreadsheet  environment.  A  description  of  the  spreadsheet 
model  is  available  in  the  appendices.  Below  the  surface  of  the  model,  the  spreadsheet  has 
been  enhanced  with  Visual  Basic  for  Applications  (VBA)  scripting  to  assist  in 
automating  the  analysis.  Some  segments  of  VBA  code  are  also  found  in  the  appendices. 
Finally,  the  Solver  Excel  add-in  (by  Frontline  Systems,  Inc.)  has  been  used  to  solve  the 
mathematical  program  specified  by  Equation  17  (implemented  through  the  spreadsheet). 

Based  upon  the  initial  parameter  settings  cited  in  Table  5,  the  resulting  decision 
maker  satisfaction  (overall  utility)  is  calculated  and  presented  in  Figure  10.  The  range  on 
the  horizontal  axis  spans  from  a  design  cost  of  $0.00  to  the  design  cost  ceiling  of 
approximately  $2,620,000.  The  design  ceiling  is  calculated  by  detennining  the  cost  of 
meeting  the  objective  level  for  each  of  the  technical  measures  of  performance,  all  within 
the  first  increment.  Because  the  risk  factor  are  the  highest  in  the  first  increment,  the 
resulting  cost  is  much  greater  than  if  the  development  was  allowed  to  progress  and  take 
advantage  of  the  lower  risk  found  in  the  latter  increments. 
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Figure  10.  Overall  Utility  -  Baseline  Configuration  (Table  5). 


As  Figure  10  indicates,  at  the  far  left  of  the  cost  range  ($0.00),  the  overall  utility 
is  approximately  0.10.  This  value  corresponds  to  the  scaling  factor  selected  for  the 
economic  MOP.  In  Chapter  III  it  was  stated  that  the  scaling  factors  equate  to  the  overall 
utility  for  the  system  when  a  given  MOP  is  at  its  best  level  and  all  others  are  at  their 
worse.  Thus,  when  no  money  is  spent  on  developing  the  system,  there  is  no  attainment 
for  the  technical  MOPs  and  their  resulting  single  attribute  utility  is  zero  (0).  Conversely, 
when  no  money  is  spent,  the  economic  MOP  is  at  its  best  possible  level  and  its  resulting 
single  attribute  utility  is  one  (1).  When  these  values  are  fed  into  Equation  15,  an  overall 
utility  equivalent  to  the  scaling  factor  for  system  cost  is  generated  (0.10). 

Looking  to  the  far  right  of  the  cost  range,  a  similar  phenomenon  occurs.  At  the 
cost  ceiling,  all  of  the  technical  MOPs  are  at  their  objective  levels  of  attainment.  Thus, 
their  resulting  utilities  are  equal  to  one  (1).  However,  the  opposite  holds  for  the 
economic  MOP.  At  the  cost  ceiling,  system  cost  is  at  its  threshold  level  and  equals  zero 
(0).  The  resulting  overall  utility  is  approximately  0.88.  From  a  heuristic  standpoint,  one 
would  reason  that  with  eight  of  nine  MOPs  at  their  highest  utility  and  the  remaining  one 
at  its  worst  (given  that  the  decision  maker  is  equally  willing  to  trade-off  each  of  the 
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MOPs),  the  resulting  overall  utility  would  be  approximately  8/9  or  0.889.  Thus  the 
model  appears  to  be  consistent  with  the  heuristic. 

Based  upon  this  analysis  of  the  end  points  of  the  cost  range  it  can  be  inferred  from 
CAIV/EA  model  that  the  overall  utility  for  a  system  will  never  be  less  than  the  value  of 
the  single  attribute  scaling  factor  for  the  economic  MOP.  Additionally,  it  will  never  be 
possible  to  have  an  overall  utility  equal  to  one  (1).  This  observation  is  attributed  to  the 
relationship  between  the  technical  MOPs  and  the  economic  MOP.  As  the  value  of  the 
technical  MOPs  increases,  the  value  of  the  economic  MOP  decreases,  and  vice  versa. 
Thus  the  trade  space  for  overall  utility  exists  between  value  for  the  economic  MOP  and 
some  value  less  than  one  (1). 


As  the  system  cost  increases  the  overall  utility  for  the  system  appears  to  increase 
as  well.  Figure  1 1  presents  a  modification  to  the  chart  presented  in  Figure  10.  A 
regression  line  has  been  added  to  model  the  relationship  between  overall  utility  and  cost. 
The  regression  line  uses  the  natural  logarithm  of  cost  to  produce  an  overall  R-squared 
value  of  0.9783. 


68 


The  baseline  configuration  does  not  warrant  any  additional  analysis.  While  it  is 
possible  to  examine  the  how  the  technical  MOPs  behave  over  the  system  cost  range,  there 
is  little  value  to  that  data.  The  trade-offs  made  in  the  baseline  configuration  are  a 
function  of  the  two  parameters  mentioned  earlier:  the  values  of  the  elements  QFD  matrix, 
R ,  and  the  levels  of  Zjmax  for  each  of  the  design  attributes.  A  decision  maker  is  more 
likely  to  be  interested  in  how  their  value  functions  (detennined  by  the  UFP)  and  their 
preferences  for  trade-offs  (set  by  the  SAU  scaling  constants,  k,)  affect  the  model.  Thus, 
the  parameter  configuration  specified  in  Table  5  will  be  adjusted  to  reflect  a  decision 
maker’s  preferences. 

Table  9  specifies  a  different  parameter  setting,  reflecting  possible  decision  maker 
preferences.  As  the  table  indicates,  the  decision  maker  has  adjusted  his  tolerance  for  risk. 
Seven  of  the  eight  technical  MOPs  now  reflect  a  utility  function  that  is  risk  averse  (UFP 
=  3.00).  This  setting  indicates  that  the  decision  maker  places  a  diminishing  marginal 
return  on  increases  in  performance  for  these  MOPs.  The  decision  maker  has  a  risk 
seeking  attitude  towards  the  remaining  technical  MOP  (Sys  Load).  By  setting  the  UFP 
equal  to  0.50,  the  decision  maker  is  indicating  a  propensity  for  increasing  marginal 
returns  for  this  MOP.  Finally,  the  UFP  for  the  cost  of  the  system  has  been  decreased  to 
0.75.  Figure  12  graphically  depicts  the  effects  of  these  new  parameter  specifications  on 
the  shape  of  the  corresponding  utility  functions. 
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Table  9.  Decision  Maker  Preference  Specifications  for  C2  System 


MOP  Factors 

PFP  UFP  ki 

MOP  Name: 

1 

Sys  Admin 

1.00 

3.00 

0.10 

2 

HMI 

1.00 

3.00 

0.10 

3 

Surveillance 

1.00 

3.00 

0.10 

4 

Identification 

1.00 

3.00 

0.10 

5 

W  &  BM* 

1.00 

3.00 

0.30 

6 

Data  Links* 

1.00 

3.00 

0.30 

7 

Trng/Sim 

1.00 

3.00 

0.10 

8 

Sys  Load 

1.00 

0.50 

0.10 

9 

Cum  Cost* 

1.00 

0.75 

0.30 
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Figure  12.  Adjusted  Decision  Maker  Utility  Functions 


Table  9  shows  that  the  decision  maker  has  also  adjusted  his  willingness  to  make 
trade-offs  between  the  various  MOPs.  The  asterisks  beside  the  Weapons  &  Battle 
Management  and  Tactical  Data  Links  entries  in  Table  3  indicate  that  these  are  key 
performance  parameters  (KPP)  for  the  notional  C2  system.  Thus,  the  decision  maker  is 
less  willing  to  make  trade-offs  with  these  MOPs  (as  indicated  by  the  scaling  constant 
values  of  0.30).  Finally,  the  schedule  weighting  factors  are  adjusted  so  that  5  =  (0.5,  0.3, 
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0.2).  This  schedule  weighting  configuration  implies  that  the  decision  maker  places 
greater  emphasis  on  delivering  capability  earlier,  rather  than  later,  in  the  system 
development. 


Figure  13.  Overall  Utility  -  Adjusted  Configuration  (Table  9) 


Figure  13  displays  the  overall  utility  curve  that  results  from  the  CAIV/EA  model 
parameters  specified  by  Table  9.  The  figure  illustrates  how  as  the  design  cost  of  the 
system  begins  to  increase  the  decision  maker’s  value  of  the  design  alternative  improves 
rapidly.  However,  from  approximately  $1,048,000  to  the  cost  ceiling  of  $2,620,000  the 
overall  utility  plateaus.  This  phenomenon  is  the  result  of  the  design  cost  constraint 
specified  by  Equation  (17).  In  this  mathematical  program,  the  design  cost  must  be  less 
than  or  equal  to  the  cost  target.  The  plateau  is  caused  by  the  diminishing  marginal 
returns  for  the  technical  MOPs  when  compared  to  the  increasing  losses  in  economic 
MOP  utility  as  the  design  cost  target  is  increased.  Thus,  by  specifying  that  the  design 
cost  must  be  less  than  or  equal  to  the  cost  target  the  impact  is  a  design  cost  that  never 
increases  beyond  $1,048,000.  In  other  words,  the  resulting  payoff  in  terms  of  technical 
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performance  does  not  offset  the  payoff  in  terms  of  cost.  From  an  overall  utility 
perspective,  the  optimization  algorithm  converges  at  $1,048,000. 

To  understand  how  the  overall  utility  for  the  system  behaves  beyond  this 
design  cost,  it  is  necessary  to  modify  Equation  (17).  The  cost  constraint  is  changed  to 
require  the  design  cost  be  equal  to  the  cost  target.  This  modification  forces  the 
optimization  algorithm  to  solve  for  design  alternatives  beyond  the  convergence  point  of 
$1,048,000.  This  modification  is  depicted  by  Figure  14. 


Overall  Utility  -  Adjusted  Config 
(TC  =  TCmax) 
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Figure  14.  Modified  Overall  Utility  -  Adjusted  Configuration  (Table  9) 


Figure  14  reveals  a  similar  ramp-up  in  utility  as  was  found  in  Figure  13.  This 
ramp-up  is  also  followed  by  a  plateau  (more  on  this  to  follow).  However,  unlike  the 
previous  chart,  Figure  14  presents  a  region  of  declining  overall  utility  towards  the  end  of 
the  design  cost  range.  This  region  clearly  illustrates  the  negative  impact  upon  overall 
utility  by  increasing  the  design  cost  of  the  system. 

Closer  inspection  of  the  overall  utility  plateau  described  in  the  previous  paragraph 
reveals  that  this  area  is  not  truly  a  region  of  equivalent  overall  utilities.  Instead,  this 
region  is  really  the  peak  of  the  overall  utility  curve.  Zooming  in  on  this  region  illustrates 
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that  overall  utility  is  increasing  from  $1,048,000  to  $1,310,000  and  then  decrease  beyond 
that  point.  Figure  15  presents  the  zoomed  view  of  this  region. 


Figure  15  shows  that  the  optimization  algorithm  converges  where  the  system  cost 
is  approximately  $1,310,000.  This  is  a  greater  alternative  design  cost  than  the  one 
presented  by  the  earlier  model  formulation  where  design  cost  could  be  less  than  or  equal 
to  the  cost  target.  However,  because  system  of  constraints  is  not  equivalent  between  the 
overall  utility  curves  presented  in  Figures  13  and  14,  the  results  are  not  directly 
comparable. 

When  evaluating  the  overall  utility  curve  in  Figure  15,  it  is  important  to  keep  the 
scale  of  the  vertical  axis  in  mind.  The  variations  in  this  range  are  rather  minute  (less  than 
four  thousandths  of  overall  utility  separating  the  highest  and  lowest  points).  Therefore, 
the  practical  significance  of  the  variation  is  limited.  What  is  important  is  the  ability  to 
address  the  macro-level  trend  in  the  utility  curve.  Within  a  system  cost  range  from 
approximately  $1,048,000  to  $1,834,000,  a  decision  maker  would  not  experience  any 
major  variations  on  overall  utility  for  the  system  design  alternatives  generated  by  the 
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optimization  algorithm.  Thus,  from  a  CAIV  perspective,  a  cost  target  could  be  moved 
back  from  the  peak  (approximately  $1,310,000)  to  the  start  of  the  plateau  (at 
approximately  $1,048,000)  and  overall  decision  maker  satisfaction  would  remain 
constant.  In  making  this  decision,  some  time  should  be  spent  evaluating  how  the 
technical  measures  of  performance  score  at  the  new  cost  target.  However  before  doing 
so,  some  additional  investigation  will  be  made  into  the  overall  utility  curve  and  how  the 
technical  MOPs  behave  across  the  entire  cost  range. 


Overall  Utility  -  Adjusted  Config 
(TC  =  TCmax) 
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Figure  16.  Adjusted  Configuration  (Table  9)  Regression  Model 


Keeping  in  mind  that  one  of  the  goals  of  this  research  is  to  be  able  to  quantify  the 
functional  relationship  between  cost  and  overall  decision  maker  satisfaction,  the  overall 
utility  curve  depicted  in  Figure  14  has  been  fit  with  polynomial  regression  line.  The 
resulting  function  allows  an  analyst  to  estimate  the  rate  of  change  in  overall  utility  given 
and  incremental  change  in  the  system  cost  (i.e.,  the  first  derivative  of  the  regression  line). 
This  observation  provides  the  decision  maker  with  a  first  order  capability  to  assess  the 
impact  of  funding  volatility  upon  the  performance  of  the  system. 
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To  evaluate  the  change  in  system  performance  as  a  function  of  cost,  it  is  valuable 
to  examine  how  the  technical  MOPs  behave  across  the  entire  cost  range.  Figures  17 
through  19  present  the  relative  performance  levels  for  each  of  the  MOPs  in  the  three 
development  increments. 


Rel  Perf  lnc-1 


-♦ — SysAdmin  — ■ — HMI  Surveillance  Identification 

— W  &  BM  — ♦ — Data  Links  — I — Trng/Sim  — - — Sys  Load 


Figure  17.  Technical  Performance  -  Increment  1 
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Rel  Perf  lnc-2 


-♦ — SysAdmin  — ■ — HMI  Surveillance  Identification 

— W  &  BM  — • — Data  Links  — I — Trng/Sim  — - — Sys  Load 


Figure  18.  Technical  Performance  -  Increment  2 


Rel  Perf  lnc-3 


-♦ — SysAdmin  — ■ — HMI  Surveillance  Identification 

— W  &  BM  — ♦ — Data  Links  — I — Trng/Sim  — - — Sys  Load 


Figure  19.  Technical  Performance  -  Increment  3 

The  technical  MOPs  appear  to  behave  in  the  expected  manner.  When  the  system 
cost  target  is  equal  to  zero  (0)  there  are  no  funds  available  to  develop  the  system,  thus  all 
of  the  technical  MOPs  have  performance  levels  of  zero  (0).  As  the  cost  target  is 
increased,  the  relative  performance  levels  for  the  technical  MOPs  increase  as  well.  Some 
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of  the  MOPs  improve  in  a  linear  manner  with  cost,  while  others  take  on  non-linear  fonns. 
Because  the  model  assumes  that  technical  performance  for  a  given  MOP  is  cumulative, 
the  relative  performance  for  the  later  increments  is  always  greater  than  or  equal  to  the 
performance  of  earlier  ones  at  a  given  cost  target.  Finally,  when  the  system  cost  target 
equals  the  cost  ceiling,  all  of  the  technical  MOPs  are  at  their  objective  level  at  the  end  of 
the  first,  core  increment.  In  reality,  there  would  be  no  need  for  the  follow-on  increments 
two  and  three. 

Returning  to  the  overall  utility  curve  depicted  in  Figure  14,  it  appeared  that  a 
plateau  in  the  function  begins  at  approximately  $1,048,000.  This  cost  target  will  be  used 
as  the  basis  for  the  remaining  portion  of  the  initial  analysis.  When  evaluating  a  single 
cost  target,  there  are  several  points  of  interest.  First,  it  is  important  to  understand  what 
the  resulting  system  perfonnance  is  at  the  specified  target.  Next,  the  phasing  of  the 
capability  delivery  is  of  interest  (i.e.,  where  in  the  development  cycle  are  the  technical 
MOPs  met).  Finally,  identification  of  the  cost  drivers  allows  for  an  appreciation  of  where 
the  budget  is  being  allocated  to  create  the  resulting  performance. 
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Relative  Performance  By  Increment 


MOP  (Cost  =$1,048,000) 


□  Inc  1  Bine  2  nine  3 


Figure  20.  Relative  Performance  by  Increment  (Cost  Target  =  $1,048,000) 

Figure  20  depicts  the  relative  performance  for  each  of  the  technical  MOPs 
generated  by  the  design  alternative  (constrained  by  a  cost  target  of  $1,048,000).  Each 
column  is  segmented  by  development  increment.  As  the  chart  indicates,  the 
preponderance  of  capability  is  delivered  in  the  first  development  increment.  This 
observation  is  consistent  with  the  schedule  weight  scheme  specified  for  the  current 
configuration  of  the  CAIV/EA  model.  Over  half  of  the  schedule  weight  is  placed  on  the 
first  increment.  Thus  a  decision  maker  would  be  pleased  to  see  that  results  from  the 
model  are  consistent  with  their  preferences.  However,  what  may  be  of  concern  are  the 
levels  of  implementation  for  the  key  performance  parameters.  While  W&BM  is  over 
75%  implemented,  Data  Links  is  less  than  60%  implemented  at  this  cost  target.  If  these 
results  are  not  acceptable,  then  the  decision  maker  may  want  to  reconsider  the  shape  of 
his  utility  function  or  his  preferences  for  trade-offs.  A  recursive  process  of  preference 
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specification  and  results  analysis  should  help  the  decision  maker  hone  in  on  a  design 
alternative  that  meets  his  requirements.  The  sensitivity  analysis  techniques  presented  in 
the  next  section  might  offer  a  means  to  decrease  the  amount  of  time  needed  to  evaluate 
the  results  of  the  CAIV/EA  model. 


Budget  Allocation  (Cost  =  $1,048,000) 


TRD 

Requirement 

Number 


Figure  21.  Cost  Driver  Identification  (Cost  Target  =  $1,048,000) 


Figure  21  addresses  the  final  portion  of  cost  target  evaluation.  The  chart  lists  the 
sixteen  design  attributes  cited  in  Table  6  (the  software  functional  requirements  described 
in  the  system’s  technical  requirements  document).  The  list  of  design  attributes  has  been 
sorted  in  descending  order  to  show  the  relative  distribution  of  the  available  budget.  This 
distribution  represents  the  design  alternative  determined  by  the  CAIV  cost  target  of 
$  1 ,048,000.  As  the  chart  reveals,  requirement  4. 1 . 1 .2  receives  almost  25%  of  the  entire 


79 


budget.  This  graphic  is  a  valuable  tool  for  understanding  which  design  attributes  are  cost 
drivers  in  the  system  development.  After  identifying  the  relevant  cost  drivers  for  a 
system  design,  a  development  IPT  should  perform  a  sanity  check  to  make  sure  that  the 
CAIV/EA  model’s  results  are  consistent  and  realistic. 

This  section  has  presented  some  initial  analysis  of  the  CAIV/EA  model  output 
data  generated  from  the  notional  C2  system.  Based  upon  this  first  round  of  analysis, 
some  questions  remain: 

•  When  overall  utility  is  equivalent,  what  is  the  resulting  trade-off  between  cost 
and  technical  perfonnance? 

•  How  do  variations  in  the  schedule  weighting  factors  affect  the  system  design 
alternative  (for  a  given  cost  target)? 

•  How  influential  are  the  single  attribute  scaling  factors  (k,)  in  affecting  the 
resulting  capability  for  a  design  alternative  (again,  for  a  given  cost  target)?, 
and 

•  What  is  the  influence  of  risk,  design  attribute  maximums  ( Zjmax ),  and  the  QFD 
matrix  ( R ),  upon  the  design  alternative? 

The  following  section  presents  additional  analysis  that  attempts  to  answer  each  of  these 
questions. 

CAIV/EA  Model  Sensitivity  Analysis 

To  evaluate  the  first  question  posed  at  the  end  of  the  previous  section,  the  end 
points  of  the  plateau  region  depicted  in  Figure  15  were  used  ($1,048,000  and  $1,843,000) 
to  specify  the  cost  targets.  Both  of  these  cost  targets  generate  an  overall  utility  of 
approximately  0.915.  Thus  by  holding  this  variable  constant,  it  is  possible  to  isolate  the 
resulting  trade-off  between  cost  and  perfonnance. 
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Figure  22.  Cost  /  Performance  Trade-offs  with  Equivalent  Overall  Utility 


Figure  22  illustrates  the  levels  of  performance  for  each  of  the  technical  MOPs  at 
the  low  and  high  ends  of  the  overall  utility  plateau.  As  is  to  be  expected,  the  lower  cost 
target  results  in  lower  levels  performance  than  the  higher  one.  The  series  entitled  “Delta” 
represents  the  difference  between  the  higher  and  lower  cost  targets’  levels  of  performance 
for  each  technical  MOP.  Based  upon  the  decision  maker’s  preferences  (as  specified  in 
table  10),  the  design  alternatives  generated  along  the  plateau  are  all  equally  satisfactory. 

In  theory,  the  less  expensive,  lesser  performing  system  is  just  as  valuable  or  satisfactory 
to  the  decision  maker  as  the  more  expensive,  higher  performing  alternative.  Thus,  the 
delta  values  describe  the  available  trade  space  between  cost  and  performance.  With  this 
is  mind,  it  is  then  necessary  for  the  decision  maker  to  review  the  relationships  depicted  in 
Figures  17  -  19  to  understand  how  performance  varies  as  a  function  of  cost  across  this 
region  of  equivalent  overall  utility.  Understanding  these  relationships  allows  the  decision 
maker  to  decide  if  variations  from  the  specified  cost  target  result  in  any  operationally 
significant  changes  to  the  system’s  performance. 
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The  next  area  of  interest  relates  to  how  variations  in  the  schedule  weighting 
factors  affect  the  system  design  alternative,  at  a  given  cost  target.  From  a  project 
management  perspective,  a  program’s  schedule  can  be  classified  as  aggressive  (seeking 
the  shortest  schedule  possible),  conservative,  or  somewhere  in  between.  Table  10  lists 
five  different  schedule  weighting  postures.  The  conservative  posture  places  all  of  the 
weighting  for  overall  utility  upon  the  incremental  utility  from  the  final  increment. 
Conversely,  the  aggressive  posture  places  all  of  its  weighting  upon  the  incremental  utility 
from  the  initial  increment. 


Table  10.  Schedule  Weighting  Factors  and  Associated  Postures 


Schedule  Weighting  Factor 

Description 

Inc  1 

Inc  2 

Inc  3 

Conserv. 

0.00 

0.00 

1.00 

Mod.  Cons. 

0.00 

0.25 

0.75 

Moderate 

0.25 

0.50 

0.25 

Mod.  Aggr. 

0.75 

0.25 

0.00 

Aggressive 

1.00 

0.00 

0.00 

As  was  defined  in  Chapter  III,  there  is  greater  development  risk  associated  with 


earlier  development  increments  than  with  later  ones.  Thus  when  cost  is  held  constant  in 


the  CAIV/EA  model,  one  would  expect  the  more  conservative  (i.e.,  longer)  development 


schedule  to  result  in  higher  levels  of  attainment  for  the  technical  MOPs  than  the 


aggressive  posture.  Figure  23  substantiates  this  assertion.  This  chart  illustrates  the  trade¬ 


offs  created  between  schedule  and  performance  when  cost  is  held  constant  (at  the 
$1,048,000  cost  target).  The  technical  MOPs  are  displayed  across  the  horizontal  axis. 


Each  MOP  cluster  contains  five  different  series,  representing  the  different  schedule 


postures  cited  in  table  10. 
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Schedule  /  Performance  Trade-offs 
Cost  Target  =  $1,048,000 
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Figure  23.  Schedule  /  Performance  Trade-offs  with  Constant  Cost  Target 


In  line  with  the  previous  assumption,  the  technical  MOPs  in  Figure  22  tend  to 
degrade  as  the  schedule  posture  progresses  from  a  conservative  to  an  aggressive 
alignment.  A  decision  maker  can  use  the  results  from  this  analysis  to  help  to  understand 
the  effect  of  accelerating  a  development  project  under  a  CAIV  constrained  budget.  From 
a  modeling  perspective,  it  is  important  that  an  analyst  correctly  captures  the  proposed  EA 
strategy  and  applies  the  appropriate  schedule  weighting  factors.  Figure  22  clearly 
illustrates  the  potential  impacts  caused  by  variations  in  these  parameters. 

As  Figure  23  demonstrates,  five  of  the  MOPs  demonstrate  a  pronounced 
degradation.  Two  (HMI  and  Identification)  stay  relatively  level  (one  slightly  decrease 
while  the  other  slightly  improves).  The  final  MOP  (W&BM)  exhibits  significant 
improvement  as  the  schedule  tightens.  This  behavior  seems  contrary  to  the  underlying 
assumption  regarding  schedule  and  performance  trade-offs.  However,  it  is  important  to 
remember  that  there  are  multiple  parameters  influencing  the  CAIV/EA  model  results  (to 
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include  the  MOP  scaling  constants  and  the  QFD  matrix).  The  following  sections  will 
address  the  influence  of  these  parameters. 


Scaling  Constant  Impact  upon  Performance 


0.10  0.20  0.30  0.40  0.50  0.60  0.70  0.80  0.90 

W&BM  Scaling  Constant 


Figure  24.  Scaling  Constant  Impact  upon  Performance  for  a  Single  MOP 


Figure  24  illustrates  the  impact  of  scaling  constant  selection  upon  the  level  of 
performance  for  a  single  MOP,  in  this  case  W&BM.  From  the  formulation  presented  in 
Chapter  III,  one  would  expect  that  as  the  value  of  the  scaling  constant  increases  in  value 
its  associated  MOP  should  improve  as  well.  The  chart  in  Figure  24  supports  this 
assertion.  For  W&BM,  as  the  value  of  the  scaling  constant  increases  from  0. 10  to  0.90, 
the  level  of  performance  tends  to  increase  as  well.  It  is  important  to  remember  that  the 
variations  in  the  W&BM  scaling  constant  are  made  while  holding  all  of  the  other  scaling 
constants  (as  described  in  table  9)  are  held  at  their  original  values  (i.e.,  the  scaling 
constants  remain  constant).  While  this  analysis  does  not  examine  the  levels  of 
performance  for  the  other  MOPs,  it  should  be  expected  that  in  a  CAIV  cost  constrained 
environment  as  W&BM  improves  the  other  technical  MOPs  degrade  (i.e.,  there  is  trade¬ 
off  incurred  by  improving  the  MOP)  .  An  analyst  must  take  care  when  eliciting  the  MOP 
scaling  constant  values  from  the  decision  maker.  A  decision  maker  should  be  aware  that 
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the  overall  system  does  not  necessarily  improve  by  artificially  inflating  the  scaling 
constant  associated  with  a  single  MOP. 


Scaling  Constant  Impact  upon  EA  Strategy 


□  Inc  3 
■  Inc  2 

□  Inc  1 
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Figure  25.  Scaling  Constant  Impact  upon  EA  Strategy  for  a  Single  MOP 


While  remaining  focused  upon  the  impact  of  scaling  constant  selection,  it  is  of 
interest  to  examine  how  the  EA  strategy  is  impacted.  Again,  from  the  CAIV/EA 
formulation  specified  in  Chapter  III,  one  would  expect  that  as  the  scaling  constant  for  a 
given  technical  MOP  increases  in  value,  the  proportion  of  capability  delivered  earlier  in 
the  development  cycle  will  increase  and  the  proportion  delivered  later  in  the  development 
will  decrease.  Figure  25  supports  this  interpretation.  The  chart  illustrates  how  as  the 
value  for  the  W&BM  scaling  constant  increases,  the  proportion  of  capability  for  the  MOP 
delivered  in  earlier  increments  generally  increases  while  the  proportion  delivered  in  later 
increments  generally  decreases.  Just  as  with  the  previous  example,  the  other  scaling 
constants  are  not  changed  during  the  analysis  (only  W&BM  is  modified).  Thus,  from  a 
CAIV/EA  trade-off  perspective  it  must  be  expected  that  as  the  delivery  schedule  for  one 
MOP  improves  there  are  other  MOPs  that  are  delayed  and  delivered  later.  The  same 
warnings  to  the  analyst  and  the  decision  maker  mentioned  previously  hold  in  this  case  as 
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well.  Inflating  the  scaling  constant  to  improve  the  delivery  for  one  MOP  implicitly 
degrades  the  delivery  of  one  or  more  of  the  remaining  MOPs. 

The  final  questions  addressed  in  this  section  relates  to  influence  of  risk,  design 
attribute  maximums  ( Zjmax ),  and  the  QFD  matrix  (R),  upon  the  design  alternative.  Up 
until  this  point,  we  have  been  primarily  concerned  with  the  influence  of  model 
parameters  upon  the  resulting  system  performance.  It  is  important  to  remember  that 
system  performance  is  ultimately  a  function  of  the  levels  of  the  design  attributes  (as 
specified  in  Equation  (17)).  Therefore,  to  truly  have  an  appreciation  for  how  the 
selection  of  CAIV/EA  model  parameters  influences  system  perfonnance,  it  is  important 
to  evaluate  how  the  design  attribute  selection  is  influenced  as  well.  A  regression  of 
several  design  attribute  associated  parameters  (upon  the  resulting  level  of  attainment  at 
the  end  of  the  final  EA  increment)  will  be  used  to  illustrate  their  influence.  Although  this 
analysis  is  completely  deterministic  in  nature,  regression  offers  an  efficient  means  to 
understand  the  significance  of  each  of  the  parameters. 

The  first  parameter  of  interest  is  the  relationship  between  a  given  design  attribute 
and  the  technical  MOPs.  The  technical  measures  of  performance  for  the  test  case  are 
linked  to  the  design  attributes  through  the  QFD  matrix  specified  in  table  7.  By  summing 
the  elements  for  each  row  of  the  matrix  it  is  possible  to  generate  a  value  that  describes  the 
magnitude  of  the  relationship  for  the  DA  and  the  technical  MOPs.  By  comparing  the  row 
sum  values  for  each  of  the  design  attributes,  it  is  possible  to  detennine  on  a  relative  scale 
which  has  the  strongest  relationship  with  technical  MOPs  and  which  has  the  weakest. 
Thus,  the  row  sum  of  the  QFD  matrix  for  a  given  DA  will  be  used  as  an  independent 
variable  in  the  regression  that  follows  shortly. 
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The  next  parameter  to  be  evaluated  is  the  vector  describing  the  design  attribute 
maximums  (Zjmax).  Because  all  of  the  design  attributes  in  the  test  case  are  described  in 
tenns  of  source  lines  of  code  (SLOC),  it  is  possible  to  draw  one-to-one  comparisons 
between  each.  Those  design  attributes  with  larger  maximum  values  may  draw  upon  more 
resources  than  those  with  smaller  maximums.  The  design  attributes  with  lower 
maximums  might  improve  overall  system  perfonnance  more  rapidly  than  those  with 
larger  maximum  values.  Therefore,  to  understand  the  roles  of  this  parameter,  it  too  will 
be  included  as  an  independent  variable  in  the  regression. 

The  final  parameter  to  be  evaluated  is  the  matrix  describing  the  design  attribute 
risk  factors  (table  8).  Because  risk  is  quantified  as  a  unit-less  value  that  influences  the 
realized  level  of  attainment  for  a  given  design  attribute  in  a  given  increment,  it  is  also 
possible  to  draw  one-to-one  comparisons.  To  generate  a  single,  composite  risk  value  for 
each  design  attribute,  the  product  of  the  risk  factors  for  each  increment  is  taken.  Those 
design  attributes  with  larger  risk  factors  may  draw  upon  more  resources  than  those  with 
smaller  risk  factors.  The  design  attributes  with  lesser  risk  factors  might  improve  overall 
system  performance  more  economically  in  earlier  development  increments  than  those 
with  larger  maximum  values.  Thus,  the  composite  risk  factor  is  included  as  an 
independent  variable  in  the  regression. 

The  dependent  variable  selected  for  the  regression  is  the  relative  level  of 
attainment  for  the  design  attributes  at  the  end  of  the  third  development  increment.  While 
any  of  the  three  increments  could  be  examined,  this  analysis  has  chosen  to  simply 
examine  the  resulting  level  of  attainment  occurring  at  the  completion  of  the  EA 
development  cycle.  Preliminary  analysis  reveals  that  taking  the  natural  logarithm  of  the 
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dependent  variable  improves  the  inferential  results  of  the  regression  model  (this  is  a 
common  transformation  used  in  linear  regression).  Therefore,  this  transformation  has 
been  used  in  the  ensuing  analysis.  Finally,  a  fourth  independent  variable  has  been  added 
to  account  for  the  interaction  between  design  attribute  maximums  and  their 
corresponding  QFD  matrix  row  sum. 


Table  11.  Design  Attribute  Parameter  Regression  Data. 


j 

TRD  Req# 

Inc  3  Rel  DA 

ln(lnc  3  Rel  DA) 

SLOC 

Row  Sum 

Comp  Risk 

SLOC  *  Row  Sum 

1 

4.1. 1.1 

0.0264 

-3.6339 

2800 

13 

0.0000066 

36400 

2 

4.1. 1.2 

0.7472 

-0.2914 

2800 

24 

0.0005540 

67200 

3 

4.1. 1.3 

1 .0000 

0.0000 

700 

13 

0.0000255 

9100 

4 

4.1. 1.4 

1 .0000 

0.0000 

700 

18 

0.0222943 

12600 

5 

4.1. 3.1 

0.1742 

-1.7476 

2700 

22 

0.0035762 

59400 

6 

4.1. 3.2 

0.0847 

-2.4684 

2700 

17 

0.0002225 

45900 

7 

4.1. 3.3 

0.0588 

-2.8330 

2700 

16 

0.0000590 

43200 

8 

4. 1.3.4 

0.0985 

-2.3172 

2700 

21 

0.0000000 

56700 

9 

4. 1.3. 5 

1 .0000 

0.0000 

720 

20 

0.0000000 

14400 

10 

4. 1.3.6 

1 .0000 

0.0000 

540 

17 

0.0003935 

9180 

11 

4. 1.3. 7 

1 .0000 

0.0000 

540 

12 

0.0000280 

6480 

12 

4.1. 3.8 

1 .0000 

0.0000 

540 

13 

0.0002260 

7020 

13 

4.1. 3.9 

1 .0000 

0.0000 

540 

26 

0.0057422 

14040 

14 

4.1.3.10 

1 .0000 

0.0000 

540 

32 

0.0001549 

17280 

15 

4.1.3.11 

1 .0000 

0.0000 

540 

29 

0.0003982 

15660 

16 

4.1.3.12 

1 .0000 

0.0000 

540 

12 

0.0000001 

6480 

Table  1 1  presents  the  data  used  in  the  ensuing  regression.  The  shaded  column 


represents  the  natural  log  transformed  increment  three  relative  design  attribute  level  data, 


used  as  the  dependent  variable.  The  four  columns  to  the  right  of  the  shaded  column 


contain  the  data  for  the  independent  variables:  design  attribute  maximums  (SLOC),  the 


row  sum  of  the  QFD  matrix  for  the  given  DA,  the  design  attribute’s  composite  risk 


factor,  and  the  interaction  term. 
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Table  12.  Design  Attribute  Regression  Results 


|  Regression  Statistics 

Multiple  R 

0.97675 

R  Square 

0.95404 

Adjusted  R  Square 

0.93733 

Standard  Error 

0.32184 

Observations 

16 

AN  OVA 

df 

SS 

MS 

F 

Significance  F 

Regression 

4 

23.65141 

5.91285 

57.08258 

0.00000 

Residual 

11 

1.13943 

0.10358 

Total 

15 

24.79084 

Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

1.82262 

0.41146 

4.42962 

0.00101 

0.91700 

2.72824 

SLOC 

-0.00316 

0.00033 

-9.58288 

0.00000 

-0.00389 

-0.00244 

Row  Sum 

-0.06495 

0.02003 

-3.24281 

0.00783 

-0.10903 

-0.02087 

Comp  Risk 

3.63403 

15.06831 

0.24117 

0.81386 

-29.53112 

36.79919 

SLOC  *  Row  Sum 

0.00011 

0.00002 

6.71255 

0.00003 

0.00008 

0.00015 

Table  12  presents  the  results  generated  by  the  regression  of  the  four  independent 


variables.  Because  this  regression  is  a  product  of  deterministically  generated  data,  the 


analysis  will  not  spend  a  large  amount  of  time  reviewing  the  statistical  underpinnings. 


Instead,  the  data  presented  in  Table  12  is  used  to  detennine  if  the  parameters  influence 


the  level  of  attainment  for  the  design  attributes,  and  if  so,  which  ones.  The  F-statistic 


value  of  57.08  (p-value  «  0.05)  indicates  that  the  parameters  (i.e.  the  independent 


variables)  have  an  influence  upon  the  dependent  variable.  In  fact,  the  significance  of  this 


value  reveals  that  these  variables  have  a  very  strong  influence  on  determining  the  level  of 


attainment. 


Knowing  that  these  parameters  play  an  important  role,  it  is  also  essential  to 
understand  which  are  most  influential.  The  p-values  for  the  individual  model  effects  are 
reviewed  to  assist  with  this  determination.  The  design  attribute  maximum  variable 
(SLOC)  and  the  interaction  tenn  (SLOC  *  Row  Sum)  have  parameter  estimates  with  the 
smallest  p-values.  Therefore,  these  parameters  have  an  influential  role  in  affecting  the 
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levels  of  the  design  attributes.  The  QFD  matrix  row  sum  variable  has  a  slightly  larger  p- 
value,  but  is  still  very  significant.  Finally,  the  composite  risk  variable  has  a  very  high  p- 
value.  Thus,  it  can  be  inferred  from  this  regression  that  the  risk  variable  is  not  as 
influential  as  the  others. 

From  a  modeling  perspective,  an  analyst  should  use  the  results  from  this 
regression  to  emphasize  specific  areas  when  formulating  a  CAIV/EA  model.  Special 
attention  should  be  paid  to  evaluating  the  design  attribute  vector  maximum  values. 
Additionally,  the  analyst  should  work  closely  with  the  development  IPT  to  carefully 
generate  the  values  to  be  used  in  the  QFD  matrix. 


90 


V.  Conclusions 


Overview 

This  chapter  begins  with  a  review  of  the  research  questions  and  objectives 
presented  in  Chapter  I.  It  then  demonstrates  how  the  methodology  and  results  presented 
in  Chapters  III  and  IV  satisfy  these  questions  and  objectives.  Next,  an  overarching 
process  is  presented  to  assist  a  development  IPT  with  incorporating  the  CAIV/EA  model 
into  their  acquisition  planning  activities.  Finally,  some  limitations  of  the  CAIV/EA 
model  are  discussed  and  some  areas  requiring  future  investigation  are  presented. 

Accomplishment  of  Research  Objectives  and  Questions 

Chapter  I  posed  the  following  question:  “Is  it  possible  to  develop  a  process  that 
integrates  CAIV  objectives  with  the  EA  framework?”  If  possible,  such  a  process  would 
help  a  user  accomplish  the  following  objectives: 

•  Better  allocation  of  constrained  resources, 

•  More  efficient  response  to  fluctuations  in  program  funding,  and 

•  Assist  planning  for  future  development  activities  (i.e.,  increments). 

Pursuant  to  these  objectives,  the  following  questions  were  raised: 

1 .  How  might  one  generate  and  graphically  depict  the  relationship  between  system 
cost  and  performance  for  a  defense  program? 

2.  What  is  the  marginal  benefit  (or  detriment)  to  a  weapon  system’s  performance 
given  an  increase  (or  decrease)  in  funding  beyond  a  cost  objective? 

3.  How  might  one  optimally  allocate  resources  across  a  program  planning  horizon 
spanning  several  increments? 
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Through  the  use  of  several  analytical  techniques,  this  research  has  endeavored  to 
logically  integrate  the  characteristics  of  CAIV  and  EA  into  a  single,  unified  mathematical 
model.  The  mathematical  program  specified  in  Equation  (17)  provides  a  rigorous 
approach  to  conducting  CAIV  cost/perfonnance/schedule/risk  trade-offs  in  an  EA 
environment  characterized  by  multiple  development  increments. 

Via  the  CAIV/EA  model  fonnulation  in  Chapter  III,  pertinent  data  were 
generated,  collected,  and  presented  in  Chapter  IV.  The  data  directly  responds  to  the  three 
questions  posed  above.  The  various  charts  and  figures  clearly  illustrate  how  the  outputs 
from  the  CAIV/EA  model  can  show  the  functional  relationship  between  a  system’s 
performance  and  its  cost.  By  incrementally  varying  the  cost  of  the  system,  it  is  possible 
to  use  the  CAIV/EA  model  to  estimate  how  the  various  measure  of  performance  will 
respond  to  these  changes.  Finally,  through  the  use  of  utility  theory  and  optimization 
techniques  it  is  possible  to  formulate  a  resource  allocation  scheme  that  translates  the 
desired  cost  target  into  a  system  design  alternative  which  satisfies  the  user. 

The  CAIV/EA  model  formulation  easily  integrates  into  a  spreadsheet 
environment.  In  fact,  all  of  the  analysis  conducted  in  Chapter  IV  was  accomplished  on  a 
standard  Microsoft  Windows  based  personal  computer  (circa  2001  technology).  This 
portability  facilitates  the  use  of  the  CAIV/EA  model  in  the  DoD  program  management 
environment.  It  is  hoped  that  by  using  the  approach  specified  in  this  research  that  more 
informed  decisions  regarding  CAIV  and  EA  are  made  (thus  meeting  the  three  goals 
specified  above). 
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Integrated  CAIV/EA  Analysis  Process 


Having  formulated  and  demonstrated  the  CAIV/EA  model  in  the  previous 
chapters,  it  is  important  to  present  a  top-level  process  that  a  development  IPT  can  use  to 
incorporate  the  model  with  their  existing  program  planning  and  analysis  processes. 
Specifically,  this  process  must  integrate  with  the  spiral  development  model  described  in 
Figure  2.  Figure  2  specifies  an  iterative  process  that  requires  risk  analysis  and  cost  / 
performance  trade-offs  in  each  revolution  of  the  development  spiral.  The  CAIV/EA 
model  integration  process  described  in  Figure  26  accomplishes  these  activities  within  the 
overarching  EA  strategy  framework. 


Figure  26.  CAIV/EA  Model  Integration  Process 
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The  goal  of  this  process  is  to  gather  the  data  required  to  build  a  CAIV/EA  model 
that  is  specific  and  relevant  to  the  current  state  of  the  weapon  system’s  development. 
Having  built  the  relevant  model,  it  can  then  be  exercised  to  generate  trade-off  data.  This 
data  is  intended  to  assist  the  ensuing  spirals’  decision  making  and  planning  activities. 
After  a  spiral  is  accomplished,  the  actual  results  from  that  activity  are  incorporated  with 
any  new  changes  to  the  model’s  parameters  to  bring  it  (the  CAIV/EA  model)  in  line  with 
the  new  development  state.  This  iterative  process  tracks  with  the  spiral  development 
process  demonstrated  in  Figure  2. 

The  CAIV/EA  model  integration  process  begins  accomplishing  three  activities  in 
tandem:  MOP  identification,  design  synthesis,  and  EA  strategy  definition.  The  first 
activity,  MOP  identification,  relates  to  defining  the  individual  technical  measures  of 
performance  for  the  weapon  system.  Metrics  are  established  for  each  MOP.  Finally, 
overall  threshold  and  objective  levels  of  performance  are  established  for  the  intended 
system  end  state. 

Design  synthesis  pertains  to  accomplishing  the  systems  engineering  activities 
necessary  to  transform  the  user’s  operational  requirements  into  definitive  system 
architecture.  Within  the  system  architecture,  design  attributes  are  identified  and 
described  in  quantitative  terms.  Finally,  alternative  solutions  to  the  design  attributes 
(where  applicable)  are  developed. 

The  EA  strategy  definition  activity  involves  specifying  parameters  associated 
with  the  overarching  EA  approach.  These  parameters  include  understanding  the  desired 
number  of  development  increments  and  the  schedule  weighting  preferences  associated 
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with  each.  Additionally,  any  incremental  threshold  or  objective  levels  of  performance 
must  be  identified. 

While  each  is  a  distinct  activity  in  its  own  right,  there  is  a  certain  degree  of 
dependency  between  the  three.  For  example,  the  system  measures  of  performance  may 
have  some  influence  on  detennining  viable  design  alternatives.  Additionally,  the 
threshold  and  objective  levels  of  performance  required  for  each  increment  in  the  EA 
strategy  are  tied  to  the  initial  definition  of  the  MOPs.  These  dependencies  are  illustrated 
by  the  dotted  lines  in  Figure  26.  The  interdependency  of  these  three  initial  activities 
reinforces  the  need  to  use  an  IPT  approach  when  integrating  the  CAIV/EA  model.  An 
analyst  should  not  expect  to  build  the  model  on  his  own.  Additionally,  no  single 
stakeholder  or  functional  area  should  dominate  any  one  of  these  activities.  Instead,  there 
should  be  strong  involvement  from  the  user,  systems  engineering,  and  program 
management  communities  at  all  times. 

Having  accomplished  the  initial  CAIV/EA  model  integration  activities,  it  is  now 
possible  to  begin  those  remaining  activities  needed  for  complete  model  formulation. 
Beginning  with  the  system’s  MOPs,  the  analyst  must  select  a  utility  function  to  model  the 
decision  maker’s  value  system  for  each  of  the  measures  of  performance  (the  test  case 
uses  the  CDF  for  the  standard  Beta  function,  but  there  are  other  alternatives).  Next, 
working  with  the  decision  maker,  the  analyst  must  elicit  a  shape  for  each  utility  function 
(in  the  test  case  this  was  accomplished  via  the  utility  function  parameter,  UFP).  Finally, 
the  overall  utility  function  must  be  synthesized  by  eliciting  the  decision  maker’s 
willingness  to  make  trade-offs  between  each  of  the  MOPs.  The  values  for  the  single 
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attribute  scaling  constants  used  in  the  overall  utility  function  are  a  result  of  this 
elicitation. 

Next,  the  activities  needed  to  translate  the  system  design  into  the  various 
measures  of  performance  must  be  accomplished.  The  QFD  matrix  linking  the  MOPs  (the 
“what’s”)  to  the  design  attributes  (the  “how’s”)  is  established.  A  QFD  relationship 
scheme  must  be  selected.  This  scheme  describes  the  numerical  basis  for  assessing  the 
strength  of  the  relationships  between  the  MOPs  and  the  design  attributes.  Next,  each 
MOP  /  DA  pair  is  evaluated  and  its  corresponding  element  in  the  QFD  matrix  is  assigned 
a  value.  Finally,  a  performance  translation  function  must  be  selected  to  transform  the 
relative  perfonnance  generated  from  the  QFD  matrix  into  an  absolute  value  consistent 
with  the  units  of  the  MOP. 

The  system  design  is  then  evaluated  from  a  cost  estimation  perspective.  Each  of 
the  design  attributes  is  reviewed  and  an  appropriate  cost  estimating  methodology  or 
relationship  is  applied  to  each.  The  level  of  cost  detail  required  from  CAIV/EA  model 
will  dictate  how  the  resulting  design  alternative  cost  is  calculated.  In  some  situations  it 
may  only  be  necessary  to  take  into  consideration  the  direct  costs  associated  with  the 
design  attributes.  In  other  instances,  the  analyst  may  decide  to  include  other  “indirect”  or 
“below  the  line”  costs  as  well.  Regardless,  it  is  important  that  a  single,  consistent  cost 
ceiling  be  calculated  for  the  system.  This  cost  ceiling  is  used  in  formulating  the  utility 
function  for  the  economic  MOP. 

Finally,  each  of  the  design  attributes  must  be  evaluated  for  their  associated 
technical  risk.  These  evaluations  are  used  to  populate  the  elements  of  the  incremental 
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risk  matrix.  The  basis  for  this  evaluation  should  be  agreed  upon  by  the  members  of  the 
IPT  and  remain  as  consistent  as  possible  during  the  development  cycle. 

It  is  now  possible  to  synthesize  all  of  the  data  and  parameters  collected  during 
these  initial  steps  into  a  single  model.  The  analysts  should  determine  the  appropriate 
platform  for  the  modeling  (the  test  case  used  Microsoft  Excel  2000®).  Additionally,  an 
optimization  algorithm  or  application  is  required  to  detennine  the  optimal  design 
alternatives.  It  may  also  be  necessary  to  use  some  degree  of  automation  or  scripting  to 
assist  with  the  model  execution  (the  test  case  used  Microsoft  Visual  Basic  for 
Applications®). 

Having  completely  formulated  and  implemented  the  model,  it  is  now  possible  to 
extract  the  data  needed  to  assist  with  the  development’s  cost/perfonnance/schedule  trade¬ 
off  decisions.  Chapter  IV  presented  several  candidate  data  products  (overall  utility, 
performance  as  a  function  of  cost,  etc.).  However,  an  analyst  should  determine  what  data 
is  required  by  the  decision  maker  and  tailor  the  data  products  accordingly.  Within 
Chapter  IV  there  are  several  examples  of  sensitivity  analysis.  The  analyst  should  conduct 
sufficient  “what-if’  analysis  to  help  illustrate  the  influence  of  the  decision  maker’s 
preferences  (as  well  as  other  model  parameters)  upon  the  resulting  system  alternative. 

Following  the  execution  of  the  development  activity,  the  “real  world”  data  should 
be  collected  and  used  to  recalibrate  the  CAIV/EA  model.  Such  data  might  entail  the  true 
level  of  attainment  for  each  of  the  design  attributes  and  the  true  level  of  performance  for 
each  of  the  MOPs.  There  might  be  changes  in  the  decision  maker’s  valuation  of  the 
different  MOPs  as  well.  In  short,  all  of  the  CAIV/EA  model  parameters  and  inputs  must 
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be  constantly  evaluated  to  help  maintain  the  relevance  of  the  model.  In  doing  so,  the  data 
generated  should  retain  its  value  to  the  decision  making  process. 

CAIV/EA  Model  Limitations 

While  this  research  has  endeavored  to  create  as  robust  and  general  of  a  model  as 
possible,  there  are  some  inherent  limitations.  This  section  will  attempt  to  address  the 
major  limitations.  Additionally,  some  recommendations  for  further  investigation  will  be 
presented.  It  is  imagined  that  many  of  these  limitations  might  be  resolved  through  minor 
modifications  to  the  current  formulation. 

Of  greatest  concern  is  the  strict  detenninistic  nature  of  the  CAIV/EA  model.  The 
formulation  as  presented  in  Chapter  III  does  not  provide  any  opportunity  to  account  for 
uncertainty  in  the  model’s  parameters.  Unfortunately,  this  limitation  is  not  consistent 
with  the  basic  nature  of  the  model.  This  model  is  intended  to  be  used  assist  the 
development  planning  of  unique,  military  focused  systems.  Because  these  systems  do  not 
yet  exist,  the  characteristics  of  their  design  and  the  risk  associated  with  their  development 
must  be  estimated.  Additionally,  the  methodologies  used  to  estimate  the  cost  of  the 
design  alternatives  are  also  based  upon  estimates.  Thus,  the  current  CAIV/EA  model 
should  be  adapted  to  address  the  uncertainty  surrounding  these  estimates.  A  Monte  Carlo 
simulation  approach  might  be  integrated  to  resolve  this  limitation.  Such  an  approach 
would  be  an  improvement  upon  the  model’s  current  implementation  of  risk.  Instead  of 
explicitly  addressing  risk  through  the  incremental  risk  matrix,  it  would  be  implicitly 
incorporated  via  the  variance  estimates  of  the  input  parameters  (specifically  the  design 
attribute  maximums). 
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Another  general  limitation  of  the  model  is  its  “development-centric”  emphasis. 

As  it  stands,  the  CAIV/EA  model  does  not  integrate  any  life-cycle  cost  factors  (i.e., 
maintenance,  operations  and  supports,  etc)  into  its  formulation.  Instead,  the  model 
focuses  solely  on  only  those  costs  associated  with  developing  the  design.  History  dictates 
that  the  preponderance  of  resources  spent  on  a  weapon  system  occur  after  it  has  been 
fielded  and  while  it  is  being  sustained.  Thus,  the  model  should  be  expanded  to  attempt  to 
capture  the  impacts  of  a  design  alternative  upon  not  just  its  development  cost,  but  also  its 
production  and  operational  support  cost.  This  expanded  model  would  seek  to  balance  the 
overall  life-cycle  cost  with  the  decision  maker’s  perceived  value  of  the  system 
performance  (as  opposed  to  simply  balancing  the  benefits  with  the  development  costs). 

Of  final  concern  is  the  manner  in  which  performance  levels  are  translated  from 
the  attained  levels  for  the  design  attributes.  The  current  fonnulation  translates  a  relative 
level  of  performance  from  the  relative  levels  of  attainments  for  the  design  attributes. 

This  translation  is  accomplished  via  the  QFD  matrix.  The  fonnulation  found  in  Chapter 
III  uses  the  strength  of  the  relationships  between  the  MOP/DA  pairs  as  the  basis  for  the 
translation.  However,  the  translation  does  not  account  for  the  conelations  between  the 
each  of  the  design  attributes.  Traditionally,  these  correlations  compose  the  “roof’  of  the 
house  of  quality.  In  some  situations,  improvements  in  one  design  attribute  might 
implicitly  improve  another  design  attribute.  Conversely,  increase  in  a  given  design 
attribute  might  degrade  the  level  of  another  design  attribute.  Fung  et  al.  (2002)  present  a 
candidate  approach  for  addressing  these  correlations  and  using  them  to  optimize  design 
selection.  This  methodology  might  be  considered  to  improve  the  quality  of  trade-offs 
made  by  the  CAIV/EA  model. 
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Summary 


This  chapter  has  demonstrated  how  the  objectives  stated  at  the  beginning  of  the 
research  have  been  met.  A  process  has  been  developed  that  clearly  incorporates  the  goals 
of  CAIV  analysis  into  an  EA  framework.  Finally,  the  known  limitations  of  the  CAIV/EA 
model  have  been  addressed  and  recommendations  for  improvements  have  been 
presented. 

Often,  DoD  acquisition  directives  are  issued  using  broad,  subjective  terms  with 
little  guidance  to  assist  their  “real  world”  implementation.  The  result  is  crippling 
confusion  resulting  from  acquisition  professionals  knowing  “what”  they  are  supposed  to 
do,  but  not  knowing  “how”  to  do  it.  This  characterization  is  accurate  for  both  CAIV  and 
EA. 

While  the  approach  described  in  this  research  is  not  panacea,  it  does  provide  the 
DoD  acquisition  community  (i.e.,  users,  program  managers,  cost  analysts,  etc.)  with  a 
disciplined,  quantitative  method  to  satisfy  the  spirit  of  the  USD(AT&L)  direction  on 
CAIV  and  EA  plans.  Additionally,  the  work  goes  a  step  further  by  identifying  a 
technique  for  integrating  the  two  initiatives.  In  other  words,  this  research  recognizes  the 
interdependent  relationships  between  program  forces  (i.e.,  cost,  schedule,  performance, 
and  risk)  and  attempts  to  rigorously  trade-off  these  elements  to  optimize  overall  user 
satisfaction.  In  short,  the  method  presented  herein  is  an  answer  to  the  question  of  “how” 
to  implement  and  integrate  CAIV  and  EA.  By  adopting  this  approach,  it  is  hoped  that 
better  acquisition  decisions  are  made,  resources  are  allocated  more  efficiently,  and  the 
user  receives  an  operationally  effective  and  suitable  system. 
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Appendix  A.  CAIV/EA  Model  Spreadsheet  Implementation  (Screen  shots) 
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A 

B 

C 

D 

E 

F 

G 

H 

1 

J 

K 

1 

2 

Increment  1 

Name:  Rel  Perf  PFP  Thrsh  Obj  Abs  Pef  UFP  Util-Raw  ki  Util-Scaled 

MOP(i) 

3 

1 

Sys  Admin 

0.71 

1.00 

0.00 

1.00 

0.71 

1.00 

0.71 

0.10 

0.95 

4 

2 

HMI 

0.76 

1.00 

0.00 

1.00 

0.76 

1.00 
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0.10 

0.95 

5 

3 
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0.00 

1.00 

0.49 

1.00 
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0.10 
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6 

4 

Identification 
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0.95 

7 

5 

W&BM 

0.57 
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1.00 

0.57 
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0.89 

8 

6 

Data  Links 

0.51 

1.00 
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0.30 
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9 

7 

Trng  /  Sim 

0.88 
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0.94 

10 

8 

Sys  Load 
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11 
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0.00 
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12 
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13 
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Increment  2 
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21 

5 

W&BM 

0.64 

1.00 

0.00 

1.00 

0.64 

1.00 

0.64 

0.30 

0.87 
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Increment  3 
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Appendix  B.  VBA  Code  Segments 


Solver  Module 

Option  Explicit 
Option  Base  1 

Dim  IncPerfO  As  Double,  Utility ()  As  Double,  Nines  As  Integer,  NMOP  As  Integer 

Sub  Solve r_Module_Main ( ) 

Dim  i  As  Integer 
RunParametersForm. Show 

Call  DeleteRangeNames  'Reset  the  named  ranges 

Call  NameRanges  'Name  the  various  ranges  used  in  the  model 

Call  ResetDecVar  'Resest  the  contents  of  the  decision  variable  cell 

For  i  =  1  To  NPoints  +  1 

Call  UpdateCostTarget (i) 

Call  SolverRoutine  'Call  the  routine  to  optimize  the  model 
Call  CollectData (i)  'Collect  the  pertinent  data 
Next  i 

Call  PrintData 
End  Sub 

Public  Sub  NameRanges () 

With  ActiveWorkbook 

With  Worksheets ( "Model" ) 

Range ( "K5" ). Name  =  "Utility" 

Range ( "K12 "). Name  =  "CostCeiling" 

Range ("K13") .Name  =  "CostTarget" 

Range ( "K14 "). Name  =  "DesignCost" 

With  Range ( "A1 " ) 


Range ( .Offset (5, 

7)  , 

•Offset (5,  7) 

. End (xlDown) ). Name  =  "] 

ci 

" 

Range ( .Offset (65, 

3)  , 

.Offset (65, 

3) .End(xlDown) ) .Name  = 

"DesignTotal" 

Range ( .Offset (65, 
' Increment  1 

4)  , 

.Offset (65, 

id  . 

End (xlDown) ) . 

Name  = 

- 

"QFDMatrix" 

Range ( .Offset (65, 

13) 

,  .Offset(65, 

13) 

. End (xlDown)  ) 

.  Name 

= 

"InclPlan" 

Range ( .Offset (65, 

14) 

,  .Offset(65, 

14) 

. End (xlDown)  ) 

.  Name 

= 

" InclRisk" 

Range ( .Offset (65, 

16) 

,  .Offset(65, 

16) 

. End (xlDown)  ) 

.  Name 

= 

"InclTotal" 

Range ( .Offset (65, 

25) 

,  .Offset(65, 

25) 

. End (xlDown)  ) 

.  Name 

= 

" InclRelDA" 

Range ( .Offset (65, 
' Increment  2 

28) 

,  .Offset(65, 

28) 

. End (xlDown)  ) 

.  Name 

= 

" InclDACost 

Range ( .Offset (65, 

17) 

,  .Offset(65, 

17) 

. End (xlDown)  ) 

.  Name 

= 

"Inc2Plan" 

Range ( .Offset (65, 

18) 

,  .Offset(65, 

18) 

. End (xlDown)  ) 

.  Name 

= 

" Inc2Risk" 

Range ( .Offset (65, 

20) 

,  .Offset(65, 

20) 

. End (xlDown)  ) 

.  Name 

= 

" Inc2Total" 

Range ( .Offset (65, 

26) 

,  .Offset(65, 

26) 

. End (xlDown)  ) 

.  Name 

= 

"Inc2RelDA" 

Range ( .Offset (65, 
' Increment  3 

29) 

,  .Offset(65, 

29) 

. End (xlDown)  ) 

.  Name 

= 

" Inc2DACost 

Range ( .Offset (65, 

21) 

,  .Offset(65, 

21) 

. End (xlDown)  ) 

.  Name 

= 

" Inc3Plan" 

Range ( .Offset (65, 

22) 

,  .Offset(65, 

22) 

. End (xlDown)  ) 

.  Name 

= 

" Inc3Risk" 

Range ( .Offset (65, 

24) 

,  .Offset(65, 

24) 

. End (xlDown)  ) 

.  Name 

= 

"Inc3Total" 

Range ( .Offset (65, 

27) 

,  .Offset(65, 

27) 

. End (xlDown)  ) 

.  Name 

= 

"Inc3RelDA" 

Range ( .Offset (65, 
With 

30) 

,  .Offset(65, 

30) 

. End (xlDown)  ) 

.  Name 

= 

" Inc3DACost 

End  With 
End  With 
End  Sub 

Sub  UpdateCostTarget (i  As  Integer) 

Select  Case  i 
Case  Is  =  0 

Worksheets ( "Model" ). Range ( "CostTarget" ) .Value  =  LowBound 
Case  Is  >  0 

Worksheets ( "Model" ). Range ( "CostTarget" ) .Value  =  _ 

LowBound  +  (i  -  1)  *  (UpBound  -  LowBound)  /  NPoints 

End  Select 
End  Sub 

Private  Sub  SolverRoutine ( ) 

Dim  DecVar  As  Range 

Set  DecVar  =  Union (Range (" InclPlan" ) ,  Range (" Inc2Plan" ) ,  Range (" Inc3Plan" ) ) 
Call  ResetDecVar 
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SolverReset 
SolverReset 
SolverOk  _ 

SetCell : =Range ( "Utility" )  ,  _ 

MaxMinVal : =1 ,  ByChange : =Union (Range (" InclPlan" ) ,  Range (" Inc2Plan" ) ,  _ 
Range (" Inc3Plan" )) ,  Engine :=1,  EngineDesc : ="Standard  GRG  Nonlinear" 
SolverAdd  _ 

CellRef : =Range ( " InclTotal" ) ,  _ 

Relation : =1 ,  FormulaText : =Range ( "DesignTotal" ) 

SolverAdd  _ 

CellRef : =Range (" Inc2Total" )  ,  _ 

Relation : =1 ,  FormulaText : =Range ( "DesignTotal" ) 

SolverAdd  _ 

CellRef : =Range (" Inc3Total" ) ,  _ 

Relation : =1 ,  FormulaText : =Range ( "DesignTotal" ) 

SolverAdd  _ 

CellRef : =Range ( "DesignCost" )  ,  _ 

Relation : =1 ,  FormulaText : =Range ( "CostTarget" ) 

SolverOptions  AssumeNonNeg : =True 
SolverSolve  UserFinish : =True 
End  Sub 

Private  Sub  CollectData (i  As  Integer) 

Dim  j  As  Integer,  k  As  Integer,  _ 
c  As  Range,  A1  As  Range 

Nines  =  3 
NMOP  =  8 

Set  A1  =  Worksheets ( "Model" ). Range ( "Al" ) 

ReDim  Preserve  IncPerf (NPoints  +  1,  NMOP,  Nines) 

ReDim  Preserve  Utility (NPoints  +  1,  Nines  +  1) 

'Capture  the  utility  data 

Utility(i,  Nines  +  1)  =  A1.0ffset(4,  10) .Value 
For  j  =  1  To  Nines 

Utility(i,  j)  =  A1.0ffset(2,  9  +  j). Value 
Next  j 

'Capture  the  relative  performance  data 
For  j  =  1  To  NMOP 

For  k  =  1  To  Nines 

IncPerf (i,  j ,  k)  =  Al. Offset (4  +  j,  1  +  k) .Value 
Next  k 
Next  j 
End  Sub 

Private  Sub  PrintDataO 

Dim  wbCAIV_EA_Data  As  Workbook,  Pathname  As  String 

Pathname  =  ThisWorkbook . Path 

Set  wbCAIV_EA_Data  =  Workbooks .Add 

wbCAIV_EA_Data . SaveAs  Filename : =Pathname  &  "\CAIV_EA_Data.xls" 

Application . DisplayAlerts  =  False 
' Worksheets ( 1 ) .Delete 
' Worksheets ( 1 ) .Delete 
Application . DisplayAlerts  =  True 

Call  PrintUtility 
Call  PrintRelPerformance 
End  Sub 

Public  Sub  FindMaxCost ( ) 

Dim  c  As  Range,  i  As  Integer 
i  =  1 

Call  ResetDecVar 

For  Each  c  In  Range (" InclPlan" ) 

c. Value  =  Range ( "DesignTotal" ). Cells (i)  /  (1  -  Range ( "InclRisk" ). Cells (i) ) 
i  =  i  +  1 
Next  c 
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Range ( "CostCeiling" ) .Value  =  Range ( "DesignCost" ) .Value 
End  Sub 

Private  Sub  FormatDataSheets ( ) 

Dim  c  As  Range,  A1  As  Range 
Set  A1  =  ActiveSheet . Range ( "Al" ) 

With  Range (Al,  Al . End (xlToRight ) ) 

.Font. Bold  =  True 
End  With 

With  Range (Al . Of f set ( 1 ,  0),  Al. Offset (1,  0 ) . End (xlDown) ) 

. NumberFormat  =  "$0.00" 

End  With 

With  Range (Al . Of f set ( 1 ,  1),  Al. Offset (1,  1 ) . End (xlDown) . End (xlToRight ) ) 
.NumberFormat  =  "0.000" 

End  With 

Al . CurrentRegion . Columns . AutoFit 
End  Sub 

Private  Sub  PrintUtility ( ) 

Dim  Al  As  Range,  i  As  Integer,  j  As  Integer,  k  As  Integer 

'Print  the  utility  data  to  the  CAIV_EA_Data  workbook 
Workbooks (2) .Worksheets (1) .Name  =  "Utility  Data" 

Set  Al  =  Worksheets ( "Utility  Data" ). Range ( "Al" ) 

'Column  headings 
For  j  =  0  To  Nines  +  1 
Select  Case  j 

Case  Is  =  0 

Al. Value  =  "Cost" 

Case  1  To  Nines 

Al. Offset (0,  j). Value  =  "Increment  "  &  j 
Case  Is  =  Nines  +  1 

Al. Offset (0,  j). Value  =  "Overall" 

End  Select 
Next  j 

'Column  contents 

For  i  =  1  To  NPoints  +  1 

Al. Offset (i,  0) .Value  =  LowBound  +  (i  -  1)  *  (UpBound  -  LowBound)  /  NPoints 
For  j  =  1  To  Nines  +  1 

A1.0ffset(i,  j). Value  =  Utility(i,  j) 

Next  j 
Next  i 

Workbooks (2) .Worksheets ( 1 ) .Columns (2) .Insert 

Workbooks (2) .Worksheets (1) .Columns ("F") .Cut  (Columns ("B") ) 

Call  FormatDataSheets 
End  Sub 

Private  Sub  PrintRelPerformance ( ) 

Dim  i  As  Integer,  j  As  Integer,  k  As  Integer,  _ 

Al  As  Range,  wsRelPerf  As  Worksheet 

Workbooks (2) .Activate 

'Create  relative  performance  data  sheets  and  dump  data  into  the  appropriate  cells 
For  i  =  1  To  Nines 

Set  wsRelPerf  =  Worksheets .Add (after : =Worksheets (Worksheets . Count) ) 
wsRelPerf . Name  =  "Rel  Perf  -  Inc  "  &  i 
Set  Al  =  ActiveSheet . Range ( "Al " ) 

'Create  the  column  headings 
For  j  =  0  To  NMOP 
Select  Case  j 
Case  Is  =  0 

Al. Value  =  "Cost" 

Case  Is  >  0 

Al.Offset(0,  j). Value  =  _ 

Workbooks (1) .Worksheets (1) .Range ("Al") .Offset (4  +  j,  1) .Value 

End  Select 
Next  j 
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'Enter  the  column  contents 

For  j  =  1  To  NPoints  +  1 

Al. Offset  (j,  0) .Value  =  LowBound  +  (j  -  1)  *  (UpBound  -  LowBound)  /  NPoints 
For  k  =  1  To  NMOP 

Al. Offset (j,  k) .Value  =  IncPerf(j,  k,  i) 

Next  k 
Next  j 

Call  FormatDataSheets 
Next  i 
End  Sub 
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